text:
# Win16 Example Application

## Table of Contents

- [Introduction](#introduction)
- [Building the Application](#building-the-application)
- [Smart Callbacks](#smart-callbacks)
- [Terms of Use](#terms-of-use)
- [Known Problems](#known-problems)
- [Changelog](#changelog)

## Introduction

This application is an example 16 bit Windows application written in C. It
accompanies the
[Building Win16 GUI Applications in C](http://www.transmissionzero.co.uk/computing/win16-apps-in-c/)
article on [Transmission Zero](http://www.transmissionzero.co.uk/).

The application runs under Windows 3.0 to Windows ME, and all versions of
Windows NT which support Win16 apps (Windows NT 3.1 to at least the x86 version
of Windows 10). It does not run under Windows 1 or 2, but there is a
[Windows 1 Version](https://github.com/TransmissionZero/Windows-1-Example-Application)
which is compatible with Windows 1 and 2 (but not compatible with Windows 9x).

The application can be built using Visual C++ 1.5x. You can
[download Visual C++ 1.52](https://my.visualstudio.com/Downloads?pid=140) if you
have a Visual Studio Subscription. If you don't have a subscription, you can
download the
[Windows Server 2003 DDK](http://download.microsoft.com/download/9/0/f/90f019ac-8243-48d3-91cf-81fc4093ecfd/1830_usa_ddk.iso)
which contains the 16 bit command line tools (most of which are from Visual C++
1.52). The 16 bit binaries can be found in "bin\bin16" of the installed DDK,
include files in "inc\inc16", and lib files in "lib\lib16". A number of the
binaries come in two versions, one with and one without a "16" suffix, e.g.
"cl.exe" and "cl16.exe". Despite the different filenames, the file contents are
identical.

The application can also be built with [Open Watcom](http://openwatcom.org/),
which is free to download.

With a few small changes to the Makefile, the application will build with
Microsoft C / C++ 7.0 and the Windows 3.1 SDK. It won't build  with the Windows
3.0 SDK because the headers don't define some of the required types, and it does
not have the "ver.h" header which is required for the version information
resource.

## Building the Application

To build the application with the Microsoft Visual C++ 1.5x GUI, go to
"project", "open", then open "Win16App.mak". To choose a debug or release build,
go to "options" then "project". The project can be built using the "build" or
"rebuild" toolbar items, or via the "project" menu.

To build the application from the command line with Visual C++ 1.5x, launch a
command prompt, run the "MSVCVARS.BAT" file which can be found in the Visual C++
"bin" directory, and then navigate to the directory containing the Makefile. Run
"nmake /f Win16App.mak" to build. This also works with the Windows Server 2003
DDK, but instead of running "MSVCVARS.BAT", you will need to add the "bin16"
directory to your "%PATH%" environment variable, "inc16" to "%INCLUDE%", and
"lib16" to "%LIB%".

To build the application in [Open Watcom](http://openwatcom.org/), open the
project up in the IDE, and choose the "Make" option from the "Targets" menu. You
can switch between debug and release builds by going to "Targets", "Target
Options", and choosing "Use Development Switches" or "Use Release Switches".

## Smart Callbacks

This application makes use of so-called "smart callbacks". This means the data segment register is loaded from the
stack segment register on entry to any callback function which is called by Windows. The result of this is that it is
not necessary to list the callback functions in the application's module definition file, and it is not necessary to use
"MakeProcInstance" to create a thunk when (e.g.) displaying a dialog box. In Visual C++ this is achieved by marking
callback functions with the "__export" modifier, and using the "/GA /GEs /GEm" compiler options (optimise far function
epilogs, load DS from SS, and increment BP on entry to the callback--required only for real mode stack walking support).
You can also use the "/GEf" option, which means all far functions are given the code to load DS from SS regardless of
whether they are callback functions. This is slightly less efficient, but does mean that callback functions don't
require the "__export" modifier.

Open Watcom does the same thing, but uses the "-zWs" compiler option to include the segment loading code.

Refer to the [Windows 1 Version](https://github.com/TransmissionZero/Windows-1-Example-Application) for an example
application which uses "MakeProcInstance" rather than smart callbacks.

## Terms of Use

Refer to "License.txt" for terms of use.

## Known Problems

The Open Watcom build of the application doesn't work correctly under Windows
3.0 when running in Real Mode. The application will start but the menu is
missing and the about dialog won't display. I've found Open Watcom to be a bit
hit and miss, where certain seemingly harmless changes of compiler option result
in an application which crashes, so it may be possible to fix this by changing
the compiler options. It runs fine under Windows 3.0 in Standard Mode and 386
Enhanced Mode.

The Open Watcom build of the application has optimisations switched off. With
optimisations enabled, the application crashes on startup.

If you have any other problems or questions, please ensure you have read this
readme file and the
[Building Win16 GUI Applications in C](http://www.transmissionzero.co.uk/computing/win16-apps-in-c/)
article. If you are still having trouble, you can
[get in contact](http://www.transmissionzero.co.uk/contact/).

## Changelog

5. 2017-05-21: Version 1.4
  - Removed unnecessary whitespace from source files.
  - Corrected min / max window size handling, which should have been using a far
    pointer for the MINMAXINFO struct.
  - Removed the original Makefile as it just duplicates the functionality of the
    Visual C++ Makefile.
  - Various updates to compiler and linker options to ensure consistency between
    Visual C++ and Open Watcom builds.

4. 2014-11-09: Version 1.3
  - Added a Makefile project for use with the Visual C++ GUI.
  - Refactored some of the code to split the source code files by functionality.

3. 2013-09-07: Version 1.2
  - Removed superfluous LOWORD() macros which had been applied to WPARAMs.

2. 2013-08-26: Version 1.1
  - Added a VERSIONINFO resource to the executable, so that version information
    can be viewed in File Manager or Windows Explorer.
  - Open Watcom build now runs in Windows 3.0 (but not in real mode).
  - Ensured all source files are no more than 8.3 characters long.

1. 2011-07-06: Version 1.0
  - First release.

Transmission Zero
2017-05-21


text:
#include "AboutDlg.h"
#include "Globals.h"
#include "MainWnd.h"
#include "Resource.h"

/* Main window class and title */
static const char MainWndClass[] = "Win16 Example Application";

/* Window procedure for our main window */
LRESULT CALLBACK MainWndProc(HWND hWnd, UINT msg, WPARAM wParam, LPARAM lParam)
{
  switch (msg)
  {
    case WM_COMMAND:
    {
      WORD id = wParam;

      switch (id)
      {
        case ID_HELP_ABOUT:
        {
          ShowAboutDialog(hWnd);
          return 0;
        }

        case ID_FILE_EXIT:
        {
          DestroyWindow(hWnd);
          return 0;
        }
      }
      break;
    }

    case WM_GETMINMAXINFO:
    {
      /* Prevent our window from being sized too small */
      MINMAXINFO FAR* minMax = (MINMAXINFO FAR*)lParam;
      minMax->ptMinTrackSize.x = 220;
      minMax->ptMinTrackSize.y = 110;

      return 0;
    }

    /* Item from system menu has been invoked */
    case WM_SYSCOMMAND:
    {
      WORD id = wParam;

      switch (id)
      {
        /* Show "about" dialog on about system menu item */
        case ID_HELP_ABOUT:
        {
          ShowAboutDialog(hWnd);
          return 0;
        }
      }
      break;
    }

    case WM_DESTROY:
    {
      PostQuitMessage(0);
      return 0;
    }
  }

  return DefWindowProc(hWnd, msg, wParam, lParam);
}

/* Register a class for our main window */
BOOL RegisterMainWindowClass()
{
  WNDCLASS wc;

  /* Class for our main window */
  wc.style         = 0;
  wc.lpfnWndProc   = &MainWndProc;
  wc.cbClsExtra    = 0;
  wc.cbWndExtra    = 0;
  wc.hInstance     = g_hInstance;
  wc.hIcon         = LoadIcon(g_hInstance, MAKEINTRESOURCE(IDI_APPICON));
  wc.hCursor       = LoadCursor(NULL, IDC_ARROW);
  wc.hbrBackground = (HBRUSH)(COLOR_BTNFACE + 1);
  wc.lpszMenuName  = MAKEINTRESOURCE(IDR_MAINMENU);
  wc.lpszClassName = MainWndClass;

  return (RegisterClass(&wc)) ? TRUE : FALSE;
}

/* Create an instance of our main window */
HWND CreateMainWindow()
{
  /* Create instance of main window */
  HWND hWnd = CreateWindowEx(0, MainWndClass, MainWndClass, WS_OVERLAPPEDWINDOW, CW_USEDEFAULT, CW_USEDEFAULT, 320, 200,
                             NULL, NULL, g_hInstance, NULL);

  if (hWnd)
  {
    /* Add "about" to the system menu */
    HMENU hSysMenu = GetSystemMenu(hWnd, FALSE);
    InsertMenu(hSysMenu, 5, MF_BYPOSITION | MF_SEPARATOR, 0, NULL);
    InsertMenu(hSysMenu, 6, MF_BYPOSITION, ID_HELP_ABOUT, "About");
  }

  return hWnd;
}

repo_language: C

mime_type: text/x-c


text:
# Adding a robot description

Before submitting a pull request, please make sure of the following:

* The corresponding description is not already in the list.
* You believe the robot description is **awesome**. This list is a curation, not a collection.
* Descriptions should come from a legal source, and be legally distributed.
* Descriptions are sorted alphabetically within each category.
* Pick the [right category](#choosing-the-right-category).
* Remove trailing whitespaces.

## Format

Use the following columns for the row of a description: ``| ROBOT-NAME | ROBOT-MAKER | [FORMAT-1](LINK-1), [FORMAT-2](LINK-2), ... | LICENSE | MESH-CHECK | INERTIA-CHECK | COLLISION-CHECK |``

* Format: can be URDF, MJCF, Xacro, etc.
* Links: point to a public repository that is, in order of preference:
    1. Up-to-date and maintained.
    2. Contains only the description (rather than a bigger project).
    3. Is the original source (rather than a fork).
* License: report the [SPDX identifier](https://spdx.org/licenses/) of the description license.
* Check the [meshes](#meshes), [inertias](#inertias) and [collisions](#collisions).

Every link and contribution, no matter how large or small, is highly appreciated and encouraged, as it helps us maintain a common repository of useful robot descriptions.

## Choosing the Right Category

Here are some rough descriptions to help you pick the right category:

- **Arms:** serial kinematic chains designed to move an end effector through a workspace.
- **Bipeds:** have two legs terminated by feet or wheels.
- **Drones:** robots that fly in the air, *a.k.a.* unmanned aerial vehicles (UAVs).
- **End Effectors:** sub-systems at the end of a limb, for example a hand at the end of an arm.
- **Mobile Manipulators:** have a wheeled mobile base and one or two arms.
- **Humanoids:** have a torso, two legs and two arms.
- **Quadrupeds:** have four legs.

## Checks

### Meshes

You can use [`yourdfpy`](https://github.com/clemense/yourdfpy/) to display a robot description. Use the ``-c`` argument to put the robot in a different configuration and check the kinematics.

### Inertias

Check that all links contain valid ``<inertial>`` tags. Also, do a quick checksum of all masses, e.g. ``cat ./some/path/robot.urdf | grep "<mass"``, and check that the sum of all values is around the expected value for the robot.

### Collisions

Check that all links contain valid ``<collision>`` tags.

## Adding a Picture to the Gallery

Optionally, you can add a picture to the gallery for a given robot description. The gallery is organized as a Markdown table of HTML linked pictures:

* Use the following cell format: ``| <img src="LINK-TO-IMAGE" alt="ROBOT-NAME" width=100> |``
* Make sure the number of cells per line stays correct. If a line is full, create a new one after it.
    * The preview feature of GitHub's editor is useful to visualize the updated table.
* The picture should be square and display well when rescaled to 100px by 100px.
* It should only depict the robot, preferably a 3D visualization over transparent background.

## Updating your PR

Making a PR adhere to the above standards can be difficult. If maintainers notice anything that needs to be improved, they will ask you to edit your PR before merging it. There's no need to open a new PR, just edit the existing one. If you're not sure how to do that, [here is a guide](https://github.com/RichardLitt/knowledge/blob/master/github/amending-a-commit-guide.md) on the different ways you can improve your PR so that it can be merged.

mime_type: text/plain


text:
# Awesome Robot Descriptions [![Awesome](https://awesome.re/badge.svg)](https://awesome.re)

A curated list of awesome robot descriptions in URDF or MJCF formats.

## Contents

* [Robot Descriptions](#robot-descriptions)
    * [Arms](#arms)
    * [Bipeds](#bipeds)
    * [Dual Arms](#dual-arms)
    * [Drones](#drones)
    * [Educational](#educational)
    * [End Effectors](#end-effectors)
    * [Mobile Manipulators](#mobile-manipulators)
    * [Humanoids](#humanoids)
    * [Quadrupeds](#quadrupeds)
    * [Wheeled](#wheeled)
* [Gallery](#gallery)
* [Related Awesome Lists](#related-awesome-lists)
* [Add a Description to the List](#add-a-description-to-the-list)

## Robot Descriptions

### Arms

| Name | Maker | Formats | License | Meshes | Inertias | Collisions |
|------|-------|---------|---------|--------|----------|------------|
| e.DO | Comau | [URDF](https://github.com/Comau/eDO_description) | BSD-3-Clause | ✔ | ✔ | ✔ |
| Gen2 (URDF) | Kinova | [URDF](https://github.com/Gepetto/example-robot-data/tree/master/robots/kinova_description) | BSD-3-Clause | ✔ | ✔ | ✔ |
| Gen3 (MJCF) | Kinova | [MJCF](https://github.com/mathieu-celerier/kinova_mj_description) | BSD-2-Clause | ✔ | ✔ | ✔ |
| Gen3 (URDF) | Kinova | [URDF](https://github.com/facebookresearch/differentiable-robot-model/tree/main/diff_robot_data/kinova_description) | MIT | ✔ | ✔ | ✔ |
| iiwa 14 | KUKA | [URDF](https://github.com/RobotLocomotion/drake/tree/master/manipulation/models/iiwa_description) | BSD-3-Clause | ✔ | ✔ | ✔ |
| M-710iC | FANUC | [URDF](https://github.com/robot-descriptions/fanuc_m710ic_description), [Xacro](https://github.com/ros-industrial/fanuc/tree/melodic-devel/fanuc_m710ic_support) | BSD-3-Clause | ✔ | ✖ | ✔ |
| Panda (MJCF) | Franka Emika | [MJCF](https://github.com/deepmind/mujoco_menagerie/tree/main/franka_emika_panda) | Apache-2.0 | ✔ | ✔ | ✔ |
| Panda (URDF) | Franka Emika | [URDF](https://github.com/Gepetto/example-robot-data/tree/master/robots/panda_description), [Xacro](https://github.com/frankaemika/franka_ros/tree/develop/franka_description) | Apache-2.0 | ✔ | ✖ | ✔ |
| Poppy Ergo Jr | Poppy Project | [URDF](https://github.com/poppy-project/poppy_ergo_jr_description) | GPL-3.0 | ✔ | ✔ | ✔ |
| Sawyer | Rethink Robotics | [Xacro](https://github.com/RethinkRobotics/sawyer_robot/tree/master/sawyer_description) | Apache-2.0 | ✔ | ✔ | ✔ |
| UR 3/5/10 | Universal Robots | [URDF](https://github.com/Gepetto/example-robot-data/blob/master/robots/ur_description/), [Xacro](https://github.com/ros-industrial/universal_robot/blob/kinetic-devel/ur_description/) | Apache-2.0 | ✔ | ✔ | ✔ |
| URe 3/5/10 | Universal Robots | [MJCF](https://github.com/deepmind/mujoco_menagerie/tree/main/universal_robots_ur5e), [URDF](https://github.com/ros-industrial/universal_robot/tree/kinetic-devel/ur_e_description) | BSD-3-Clause | ✔ | ✔ | ✔ |

### Bipeds

| Name | Maker | Formats | License | Meshes | Inertias | Collisions |
|------|-------|---------|---------|--------|----------|------------|
| Bolt | ODRI | [URDF](https://github.com/Gepetto/example-robot-data/tree/master/robots/bolt_description) | BSD-3-Clause | ✔ | ✔ | ✔ |
| Cassie (MJCF) | Agility Robotics | [MJCF](https://github.com/deepmind/mujoco_menagerie/tree/main/agility_cassie) | MIT | ✔ | ✔ | ✔ |
| Cassie (URDF) | Agility Robotics | [URDF](https://github.com/robot-descriptions/cassie_description) | MIT | ✔ | ✔ | ✔ |
| Rhea | Gabrael Levine | [URDF](https://github.com/G-Levine/rhea_description) | ✖ | ✔ | ✖ | ✔ |
| Spryped | Benjamin Bokser | [URDF](https://github.com/bbokser/spryped/tree/master/spryped_urdf_rev06) | GPL-3.0 | ✔ | ✔ | ✔ |
| Upkie | Tast's Robots | [URDF](https://github.com/tasts-robots/upkie_description) | Apache-2.0 | ✔ | ✔ | ✔ |

### Dual Arms

| Name | Maker | Formats | License | Meshes | Inertias | Collisions |
|------|-------|---------|---------|--------|----------|------------|
| Baxter | Rethink Robotics | [URDF](https://github.com/RethinkRobotics/baxter_common/blob/master/baxter_description/urdf/baxter.urdf), [Xacro](https://github.com/RethinkRobotics/baxter_common/blob/master/baxter_description/urdf/baxter.urdf.xacro) | BSD-3-Clause | ✔ | ✔ | ✔ |
| NEXTAGE | Kawada Robotics | [URDF](https://github.com/tork-a/rtmros_nextage/tree/indigo-devel/nextage_description) | BSD | ✔ | ✔ | ✔ |
| Poppy Torso | Pollen Robotics | [URDF](https://github.com/poppy-project/poppy_torso_description) | GPL-3.0 | ✔ | ✔ | ✔ |
| YuMi | ABB | [URDF](https://github.com/OrebroUniversity/yumi/tree/master/yumi_description) | BSD-2-Clause | ✔ | ✔ | ✔ |

### Drones

| Name | Maker | Formats | License | Meshes | Inertias | Collisions |
|------|-------|---------|---------|--------|----------|------------|
| Crazyflie 2.0 | Bitcraze | [URDF](https://github.com/utiasDSL/gym-pybullet-drones/tree/master/gym_pybullet_drones/assets) | MIT | ✔ | ✔ | ✔ |

### Educational

| Name | Formats | License | Meshes | Inertias | Collisions |
|------|---------|---------|--------|----------|------------|
| Double Pendulum | [URDF](https://github.com/Gepetto/example-robot-data/tree/master/robots/double_pendulum_description) | BSD-3-Clause | ✔ | ✔ | ✔ |
| FingerEdu | [URDF](https://github.com/Gepetto/example-robot-data/tree/master/robots/finger_edu_description) | BSD-3-Clause | ✔ | ✔ | ✔ |
| Simple Humanoid | [URDF](https://github.com/laas/simple_humanoid_description) | BSD-2-Clause | ✔ | ✔ | ✖ |
| TriFingerEdu | [URDF](https://github.com/facebookresearch/differentiable-robot-model/tree/main/diff_robot_data/trifinger_edu_description) | MIT | ✔ | ✔ | ✔ |

### End Effectors

| Name | Maker | Formats | License | Meshes | Inertias | Collisions |
|------|-------|---------|---------|--------|----------|------------|
| Allegro Hand | Wonik Robotics | [URDF](https://github.com/RobotLocomotion/drake/tree/master/manipulation/models/allegro_hand_description) | BSD | ✔ | ✔ | ✔ |
| BarrettHand | Barrett Technology | [URDF](https://github.com/jhu-lcsr-attic/bhand_model/tree/master/robots) | BSD | ✖ | ✔ | ✔ |
| Robotiq 2F-85 | Robotiq | [MJCF](https://github.com/deepmind/mujoco_menagerie/tree/main/robotiq_2f85), [URDF](https://github.com/a-price/robotiq_arg85_description), [Xacro](https://github.com/ros-industrial/robotiq/tree/kinetic-devel/robotiq_2f_85_gripper_visualization) | BSD-2-Clause | ✔ | ✔ | ✔ |
| Shadow Hand E3M5 | The Shadow Robot Company | [MJCF](https://github.com/deepmind/mujoco_menagerie/tree/main/shadow_hand) | Apache-2.0 | ✔ | ✔ | ✔ |
| WSG 50 | SCHUNK | [SDF](https://github.com/RobotLocomotion/drake/tree/master/manipulation/models/wsg_50_description) | BSD-3-Clause | ✔ | ✔ | ✔ |

### Mobile Manipulators

| Name | Maker | Formats | License | Meshes | Inertias | Collisions |
|------|-------|---------|---------|--------|----------|------------|
| Eve R3 | Halodi | [URDF](https://github.com/Halodi/halodi-robot-models) | Apache-2.0 | ✔ | ✔ | ✔ |
| Fetch | Fetch Robotics | [URDF](https://github.com/openai/roboschool/tree/master/roboschool/models_robot/fetch_description) | MIT | ✔ | ✔ | ✔ |
| Ginger | Paaila Technology | [URDF](https://github.com/Rayckey/GingerURDF) | BSD | ✔ | ✔ | ✔ |
| Pepper | SoftBank Robotics | [URDF](https://github.com/jrl-umi3218/pepper_description) | BSD-2-Clause | ✔ | ✔ | ✔ |
| PR2 | Willow Garage | [URDF](https://github.com/ankurhanda/robot-assets/tree/master/urdfs/robots/pr2), [Xacro](https://github.com/PR2/pr2_common/blob/melodic-devel/pr2_description/urdf/common.xacro) | BSD | ✔ | ✔ | ✔ |
| Reachy | Pollen Robotics | [URDF](https://github.com/aubrune/reachy_description) | Apache-2.0 | ✔ | ✔ | ✔ |
| Ridgeback | Clearpath Robotics | [Xacro](https://github.com/ridgeback/ridgeback_manipulation/tree/melodic-devel/ridgeback_manipulation_description) | BSD | ✔ | ✔ | ✔ |
| Stretch RE1 | Hello Robot | [URDF](https://github.com/robot-descriptions/stretch_description), [Xacro](https://github.com/hello-robot/stretch_ros/tree/master/stretch_description) | CC-BY-NC-SA-4.0 | ✔ | ✔ | ✔ |
| TIAGo | PAL Robotics | [URDF](https://github.com/Gepetto/example-robot-data/tree/master/robots/tiago_description) | CC-BY-NC-ND-3.0 | ✔ | ✔ | ✔ |

### Humanoids

| Name | Maker | Formats | License | Meshes | Inertias | Collisions |
|------|-------|---------|---------|--------|----------|------------|
| Atlas DRC (v3) | Boston Dynamics | [URDF](https://github.com/RobotLocomotion/drake/tree/master/examples/atlas) | BSD-3-Clause | ✔ | ✔ | ✔ |
| Atlas v4 | Boston Dynamics | [URDF](https://github.com/openai/roboschool/tree/1.0.49/roboschool/models_robot/atlas_description) | MIT | ✔ | ✔ | ✔ |
| Digit | Agility Robotics | [URDF](https://github.com/adubredu/DigitRobot.jl) | ✖ | ✔ | ✔ | ✔ |
| Draco3 | Apptronik | [URDF](https://github.com/shbang91/draco3_description) | BSD-2-Clause | ✔ | ✔ | ✔ |
| ergoCub | IIT | [URDF](https://github.com/icub-tech-iit/ergocub-software/tree/master/urdf/ergoCub) | BSD-3-Clause | ✔ | ✔ | ✔ |
| iCub | IIT | [URDF](https://github.com/robotology/icub-models/tree/master/iCub) | CC-BY-SA-4.0 | ✔ | ✔ | ✔ |
| JAXON | JSK | [COLLADA](https://github.com/stephane-caron/openrave_models/tree/master/JAXON), [URDF](https://github.com/robot-descriptions/jaxon_description), [VRML](https://github.com/start-jsk/rtmros_choreonoid/tree/master/jvrc_models/JAXON_JVRC) | CC-BY-SA-4.0 | ✔ | ✔ | ✔ |
| JVRC-1 | AIST | [MJCF](https://github.com/isri-aist/jvrc_mj_description/), [URDF](https://github.com/stephane-caron/jvrc_description) | BSD-2-Clause | ✔ | ✔ | ✔ |
| NAO | SoftBank Robotics | [URDF](https://github.com/ros-naoqi/nao_robot/tree/master/nao_description/urdf/naoV50_generated_urdf), [Xacro](https://github.com/ros-naoqi/nao_robot/tree/master/nao_description/) | BSD-3-Clause | [:heavy_minus_sign:](https://github.com/ros-naoqi/nao_meshes#readme) | ✔ | ✔ |
| Robonaut 2 | NASA JSC Robotics | [URDF](https://github.com/gkjohnson/nasa-urdf-robots/tree/master/r2_description) | NASA-1.3 | ✔ | ✔ | ✔ |
| Romeo | Aldebaran Robotics | [URDF](https://github.com/ros-aldebaran/romeo_robot/tree/master/romeo_description) | BSD-3-Clause | ✔ | ✔ | ✔ |
| SigmaBan | Rhoban | [URDF](https://github.com/Rhoban/sigmaban_urdf) | MIT | ✔ | ✔ | ✔ |
| TALOS | PAL Robotics | [URDF](https://github.com/stack-of-tasks/talos-data) | LGPL-3.0 | ✔ | ✔ | ✔ |
| Valkyrie | NASA JSC Robotics | [URDF](https://github.com/gkjohnson/nasa-urdf-robots/tree/master/val_description/model), [Xacro](https://gitlab.com/nasa-jsc-robotics/val_description) | NASA-1.3 | ✔ | ✔ | ✔ |
| WALK-MAN | IIT | [Xacro](https://github.com/ADVRHumanoids/iit-walkman-ros-pkg/tree/master/walkman_urdf) | BSD-3-Clause | ✔ | ✔ | ✔ |

### Quadrupeds

| Name | Maker | Formats | License | Meshes | Inertias | Collisions |
|------|-------|---------|---------|--------|----------|------------|
| A1 | UNITREE Robotics | [MJCF](https://github.com/unitreerobotics/unitree_mujoco/tree/main/data/a1/xml), [URDF](https://github.com/unitreerobotics/unitree_ros/tree/master/robots/a1_description) | MPL-2.0 | ✔ | ✔ | ✔ |
| Aliengo | UNITREE Robotics | [MJCF](https://github.com/unitreerobotics/unitree_mujoco/tree/main/data/aliengo/xml), [URDF](https://github.com/unitreerobotics/unitree_ros/tree/master/robots/aliengo_description) | MPL-2.0 | ✔ | ✔ | ✔ |
| ANYmal B | ANYbotics | [MJCF](https://github.com/deepmind/mujoco_menagerie/tree/main/anybotics_anymal_b), [URDF](https://github.com/ANYbotics/anymal_b_simple_description) | BSD-3-Clause | ✔ | ✔ | ✔ |
| ANYmal C | ANYbotics | [MJCF](https://github.com/deepmind/mujoco_menagerie/tree/main/anybotics_anymal_c), [URDF](https://github.com/ANYbotics/anymal_c_simple_description) | BSD-3-Clause | ✔ | ✔ | ✔ |
| Go1 | UNITREE Robotics | [MJCF](https://github.com/unitreerobotics/unitree_mujoco/tree/main/data/go1/xml), [URDF](https://github.com/unitreerobotics/unitree_mujoco/tree/main/data/go1/urdf) | BSD-3-Clause | ✔ | ✔ | ✔ |
| HyQ | IIT | [URDF](https://github.com/Gepetto/example-robot-data/tree/master/robots/hyq_description) | Apache-2.0 | ✔ | ✔ | ✔ |
| Laikago | UNITREE Robotics | [MJCF](https://github.com/unitreerobotics/unitree_mujoco/tree/main/data/laikago/xml), [URDF](https://github.com/unitreerobotics/unitree_ros/tree/master/robots/laikago_description) | MPL-2.0 | ✔ | ✔ | ✔ |
| Mini Cheetah | MIT | [URDF](https://github.com/Derek-TH-Wang/mini_cheetah_urdf) | BSD | ✔ | ✔ | ✔ |
| Minitaur | Ghost Robotics | [URDF](https://github.com/bulletphysics/bullet3/blob/master/data/quadruped/minitaur.urdf) | BSD-2-Clause | ✔ | ✔ | ✔ |
| Solo | ODRI | [URDF](https://github.com/Gepetto/example-robot-data/tree/master/robots/solo_description) | BSD-3-Clause | ✔ | ✔ | ✔ |
| Spot | Boston Dynamics | [Xacro](https://github.com/clearpathrobotics/spot_ros/tree/master/spot_description) | ✖ | ✔ | ✖ | ✔ |

### Wheeled

| Name | Maker | Formats | License | Meshes | Inertias | Collisions |
|------|-------|---------|---------|--------|----------|------------|
| Boxer | Clearpath Robotics | [Xacro](https://github.com/boxer-cpr/boxer/tree/noetic-devel/boxer_description) | BSD-3-Clause | ✔ | ✔ | ✔ |
| Dingo | Clearpath Robotics | [Xacro](https://github.com/dingo-cpr/dingo/tree/melodic-devel/dingo_description) | BSD-3-Clause | ✔ | ✔ | ✔ |
| Husky | Clearpath Robotics | [Xacro](https://github.com/husky/husky/tree/noetic-devel/husky_description) | BSD-3-Clause | ✔ | ✔ | ✔ |
| Jackal | Clearpath Robotics | [Xacro](https://github.com/jackal/jackal/tree/noetic-devel/jackal_description) | BSD-3-Clause | ✔ | ✔ | ✔ |
| Ridgeback | Clearpath Robotics | [Xacro](https://github.com/ridgeback/ridgeback/tree/melodic-devel/ridgeback_description) | BSD-3-Clause | ✔ | ✔ | ✔ |

## Gallery

|  |  |  |  |  |
|--|--|--|--|--|
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  | |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |
|  |  |  |  |  |

## Related Awesome Lists

* [Awesome Open Source Robots](https://github.com/tasts-robots/awesome-open-source-robots)
* [Awesome URDF](https://github.com/ami-iit/awesome-urdf)

## Add a Description to the List

New robot descriptions are welcome! Check out the [guidelines](CONTRIBUTING.md), then open a PR.


text:
name: CI
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]
jobs:
  Awesome_Lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
        with:
          fetch-depth: 0
      - run: npx awesome-lint


text:
# Audio Only
CUDA_VISIBLE_DEVICES=0 python train.py --output_dir=./save_ao/0  \
    --model_name_or_path=ctl/wav2vec2-large-xlsr-cantonese \
    --train_manifest_path=dataset/mm_train_metadata.csv \
    --valid_manifest_path=dataset/mm_valid_metadata.csv \
    --test_manifest_path=dataset/mm_test_metadata.csv \
    --num_workers=8 --preprocessing_num_workers=8 \
    --audio_column_name=audio_path --text_column_name=text_path \
    --per_device_train_batch_size=16 --per_device_eval_batch_size=16 \
    --dataloader_num_workers=32 --dataloader_pin_memory \
    --seed=0 --num_train_epochs=20 --learning_rate=5e-5 \
    --fp16 --fp16_backend=amp \
    --logging_strategy=steps --logging_steps=10 --report_to=tensorboard \
    --evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=100 \
    --save_steps=1 --save_strategy=epoch --save_total_limit=1 \
    --metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=0 python train.py --output_dir=./save_ao/1  \
    --model_name_or_path=ctl/wav2vec2-large-xlsr-cantonese \
    --train_manifest_path=dataset/mm_train_metadata.csv \
    --valid_manifest_path=dataset/mm_valid_metadata.csv \
    --test_manifest_path=dataset/mm_test_metadata.csv \
    --num_workers=8 --preprocessing_num_workers=8 \
    --audio_column_name=audio_path --text_column_name=text_path \
    --per_device_train_batch_size=16 --per_device_eval_batch_size=16 \
    --dataloader_num_workers=32 --dataloader_pin_memory \
    --seed=1 --num_train_epochs=20 --learning_rate=5e-5 \
    --fp16 --fp16_backend=amp \
    --logging_strategy=steps --logging_steps=10 --report_to=tensorboard \
    --evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=100 \
    --save_steps=1 --save_strategy=epoch --save_total_limit=1 \
    --metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=0 python train.py --output_dir=./save_ao/2  \
    --model_name_or_path=ctl/wav2vec2-large-xlsr-cantonese \
    --train_manifest_path=dataset/mm_train_metadata.csv \
    --valid_manifest_path=dataset/mm_valid_metadata.csv \
    --test_manifest_path=dataset/mm_test_metadata.csv \
    --num_workers=8 --preprocessing_num_workers=8 \
    --audio_column_name=audio_path --text_column_name=text_path \
    --per_device_train_batch_size=16 --per_device_eval_batch_size=16 \
    --dataloader_num_workers=32 --dataloader_pin_memory \
    --seed=2 --num_train_epochs=20 --learning_rate=5e-5 \
    --fp16 --fp16_backend=amp \
    --logging_strategy=steps --logging_steps=10 --report_to=tensorboard \
    --evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=100 \
    --save_steps=1 --save_strategy=epoch --save_total_limit=1 \
    --metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=0 python train.py --output_dir=./save_ao/3  \
    --model_name_or_path=ctl/wav2vec2-large-xlsr-cantonese \
    --train_manifest_path=dataset/mm_train_metadata.csv \
    --valid_manifest_path=dataset/mm_valid_metadata.csv \
    --test_manifest_path=dataset/mm_test_metadata.csv \
    --num_workers=8 --preprocessing_num_workers=8 \
    --audio_column_name=audio_path --text_column_name=text_path \
    --per_device_train_batch_size=16 --per_device_eval_batch_size=16 \
    --dataloader_num_workers=32 --dataloader_pin_memory \
    --seed=3 --num_train_epochs=20 --learning_rate=5e-5 \
    --fp16 --fp16_backend=amp \
    --logging_strategy=steps --logging_steps=10 --report_to=tensorboard \
    --evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=100 \
    --save_steps=1 --save_strategy=epoch --save_total_limit=1 \
    --metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=0 python train.py --output_dir=./save_ao/4  \
    --model_name_or_path=ctl/wav2vec2-large-xlsr-cantonese \
    --train_manifest_path=dataset/mm_train_metadata.csv \
    --valid_manifest_path=dataset/mm_valid_metadata.csv \
    --test_manifest_path=dataset/mm_test_metadata.csv \
    --num_workers=8 --preprocessing_num_workers=8 \
    --audio_column_name=audio_path --text_column_name=text_path \
    --per_device_train_batch_size=16 --per_device_eval_batch_size=16 \
    --dataloader_num_workers=32 --dataloader_pin_memory \
    --seed=4 --num_train_epochs=20 --learning_rate=5e-5 \
    --fp16 --fp16_backend=amp \
    --logging_strategy=steps --logging_steps=10 --report_to=tensorboard \
    --evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=100 \
    --save_steps=1 --save_strategy=epoch --save_total_limit=1 \
    --metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

repo_name: HLTCHKUST/CI-AVSR

repo_language: Python


text:
# Eval udio Only Noisy
CUDA_VISIBLE_DEVICES=1 python eval.py --output_dir=./save_ao_noisy/0/eval \
--model_name_or_path=save_ao/14045/checkpoint-4500  --test_manifest_path=cache_noisy/0/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=1 python eval.py --output_dir=./save_ao_noisy/1/eval \
--model_name_or_path=save_ao/14045/checkpoint-4500  --test_manifest_path=cache_noisy/1/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=1 python eval.py --output_dir=./save_ao_noisy/2/eval \
--model_name_or_path=save_ao/14045/checkpoint-4500  --test_manifest_path=cache_noisy/2/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=1 python eval.py --output_dir=./save_ao_noisy/3/eval \
--model_name_or_path=save_ao/14045/checkpoint-4500  --test_manifest_path=cache_noisy/3/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=1 python eval.py --output_dir=./save_ao_noisy/4/eval \
--model_name_or_path=save_ao/14045/checkpoint-4500  --test_manifest_path=cache_noisy/4/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=1 python eval.py --output_dir=./save_ao_noisy/5/eval \
--model_name_or_path=save_ao/14045/checkpoint-4500  --test_manifest_path=cache_noisy/5/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=1 python eval.py --output_dir=./save_ao_noisy/6/eval \
--model_name_or_path=save_ao/14045/checkpoint-4500  --test_manifest_path=cache_noisy/6/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=1 python eval.py --output_dir=./save_ao_noisy/7/eval \
--model_name_or_path=save_ao/14045/checkpoint-4500  --test_manifest_path=cache_noisy/7/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=1 python eval.py --output_dir=./save_ao_noisy/8/eval \
--model_name_or_path=save_ao/14045/checkpoint-4500  --test_manifest_path=cache_noisy/8/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=1 python eval.py --output_dir=./save_ao_noisy/9/eval \
--model_name_or_path=save_ao/14045/checkpoint-4500  --test_manifest_path=cache_noisy/9/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=1 python eval.py --output_dir=./save_ao_noisy/all/eval \
--model_name_or_path=save_ao/14045/checkpoint-4500  --test_manifest_path=cache_noisy/all/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True


text:
# Multimodal
CUDA_VISIBLE_DEVICES=1 python train.py --output_dir=./save_mm/0  \
    --model_name_or_path=ctl/wav2vec2-large-xlsr-cantonese \
    --train_manifest_path=dataset/mm_train_metadata.csv \
    --valid_manifest_path=dataset/mm_valid_metadata.csv \
    --test_manifest_path=dataset/mm_test_metadata.csv \
    --num_workers=8 --preprocessing_num_workers=8 --use_video \
    --audio_column_name=audio_path --text_column_name=text_path --video_column_name=lip_image_path \
    --per_device_train_batch_size=8 --per_device_eval_batch_size=8 --gradient_accumulation_steps 2 \
    --dataloader_num_workers=32 --dataloader_pin_memory \
    --seed=0 --num_train_epochs=20 --learning_rate=5e-5 \
    --fp16 --fp16_backend=amp \
    --logging_strategy=steps --logging_steps=10 --report_to=tensorboard \
    --evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=100 \
    --save_steps=1 --save_strategy=epoch --save_total_limit=1 \
    --metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=1 python train.py --output_dir=./save_mm/1  \
    --model_name_or_path=ctl/wav2vec2-large-xlsr-cantonese \
    --train_manifest_path=dataset/mm_train_metadata.csv \
    --valid_manifest_path=dataset/mm_valid_metadata.csv \
    --test_manifest_path=dataset/mm_test_metadata.csv \
    --num_workers=8 --preprocessing_num_workers=8 --use_video \
    --audio_column_name=audio_path --text_column_name=text_path --video_column_name=lip_image_path \
    --per_device_train_batch_size=8 --per_device_eval_batch_size=8 --gradient_accumulation_steps 2 \
    --dataloader_num_workers=32 --dataloader_pin_memory \
    --seed=1 --num_train_epochs=20 --learning_rate=5e-5 \
    --fp16 --fp16_backend=amp \
    --logging_strategy=steps --logging_steps=10 --report_to=tensorboard \
    --evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=100 \
    --save_steps=1 --save_strategy=epoch --save_total_limit=1 \
    --metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=1 python train.py --output_dir=./save_mm/2  \
    --model_name_or_path=ctl/wav2vec2-large-xlsr-cantonese \
    --train_manifest_path=dataset/mm_train_metadata.csv \
    --valid_manifest_path=dataset/mm_valid_metadata.csv \
    --test_manifest_path=dataset/mm_test_metadata.csv \
    --num_workers=8 --preprocessing_num_workers=8 --use_video \
    --audio_column_name=audio_path --text_column_name=text_path --video_column_name=lip_image_path \
    --per_device_train_batch_size=8 --per_device_eval_batch_size=8 --gradient_accumulation_steps 2 \
    --dataloader_num_workers=32 --dataloader_pin_memory \
    --seed=2 --num_train_epochs=20 --learning_rate=5e-5 \
    --fp16 --fp16_backend=amp \
    --logging_strategy=steps --logging_steps=10 --report_to=tensorboard \
    --evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=100 \
    --save_steps=1 --save_strategy=epoch --save_total_limit=1 \
    --metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=1 python train.py --output_dir=./save_mm/3  \
    --model_name_or_path=ctl/wav2vec2-large-xlsr-cantonese \
    --train_manifest_path=dataset/mm_train_metadata.csv \
    --valid_manifest_path=dataset/mm_valid_metadata.csv \
    --test_manifest_path=dataset/mm_test_metadata.csv \
    --num_workers=8 --preprocessing_num_workers=8 --use_video \
    --audio_column_name=audio_path --text_column_name=text_path --video_column_name=lip_image_path \
    --per_device_train_batch_size=8 --per_device_eval_batch_size=8 --gradient_accumulation_steps 2 \
    --dataloader_num_workers=32 --dataloader_pin_memory \
    --seed=3 --num_train_epochs=20 --learning_rate=5e-5 \
    --fp16 --fp16_backend=amp \
    --logging_strategy=steps --logging_steps=10 --report_to=tensorboard \
    --evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=100 \
    --save_steps=1 --save_strategy=epoch --save_total_limit=1 \
    --metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=1 python train.py --output_dir=./save_mm/4  \
    --model_name_or_path=ctl/wav2vec2-large-xlsr-cantonese \
    --train_manifest_path=dataset/mm_train_metadata.csv \
    --valid_manifest_path=dataset/mm_valid_metadata.csv \
    --test_manifest_path=dataset/mm_test_metadata.csv \
    --num_workers=8 --preprocessing_num_workers=8 --use_video \
    --audio_column_name=audio_path --text_column_name=text_path --video_column_name=lip_image_path \
    --per_device_train_batch_size=8 --per_device_eval_batch_size=8 --gradient_accumulation_steps 2 \
    --dataloader_num_workers=32 --dataloader_pin_memory \
    --seed=4 --num_train_epochs=20 --learning_rate=5e-5 \
    --fp16 --fp16_backend=amp \
    --logging_strategy=steps --logging_steps=10 --report_to=tensorboard \
    --evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=100 \
    --save_steps=1 --save_strategy=epoch --save_total_limit=1 \
    --metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True


text:
import os, sys
import logging
import numpy as np
import pandas as pd
import argparse

import torchaudio
import torch
import re
import json
import librosa
from datasets import load_from_disk, load_dataset, load_metric

from transformers import (
    set_seed,
    Wav2Vec2Processor,
    Wav2Vec2CTCTokenizer,
    Wav2Vec2FeatureExtractor,
    Wav2Vec2ForCTC,
    Wav2Vec2Config,
    Trainer,
    TrainingArguments,
    HfArgumentParser,
    EarlyStoppingCallback
)

from datasets import DatasetDict, load_metric, load_from_disk
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Union
import datasets
import pickle

import editdistance
import jieba
from itertools import chain

import transformers
from transformers.trainer_utils import get_last_checkpoint, is_main_process
from transformers.utils import check_min_version
from transformers.utils.versions import require_version

from args_helper import ModelArguments, DataArguments
from datasets import set_caching_enabled

from utils import CHARS_TO_IGNORE, remove_special_characters, extract_all_chars, tokenize_for_mer, tokenize_for_cer
from data_utils import speech_file_to_array_fn, load_dataset
from data_collator_ctc import DataCollatorCTCWithPadding, DataCollatorMMCTCWithPadding
from mm_wrapper import MMWav2Vec2Model

set_caching_enabled(True)
logger = logging.getLogger(__name__)

#####
# Main Functions
#####
def run(model_args, data_args, training_args):
    ###
    # Prepare Processor & Model
    ###
    print('Load Wav2Vec2 model and processor...')
    config = Wav2Vec2Config.from_pretrained('ctl/wav2vec2-large-xlsr-cantonese')
    config.update({
        "mask_time_prob": 0,
        "mask_time_length": 0,
        "mask_feature_prob": 0,
        "mask_feature_length": 0,
        "gradient_checkpointing": True,
    })
    processor = Wav2Vec2Processor.from_pretrained('ctl/wav2vec2-large-xlsr-cantonese')
    wav2vec2ctc = Wav2Vec2ForCTC(config=config)

    if data_args.use_video:
        model = MMWav2Vec2Model(wav2vec2ctc)
    else:
        model = wav2vec2ctc
    model.load_state_dict(torch.load(f'{model_args.model_name_or_path}/pytorch_model.bin'))
    model.cuda()

    print('Loading cached dataset...')
    vectorized_datasets = datasets.load_from_disk(f'{data_args.test_manifest_path}')

    if data_args.preprocessing_only:
        logger.info(f"Data preprocessing finished. Files cached at {vectorized_datasets.cache_files}")
        return

    ###
    # Prepare Data Collator and Trainer
    ###
    print('Preparing Trainer...')

    # Instantiate custom data collator
    if data_args.use_video:
        data_collator = DataCollatorMMCTCWithPadding(processor=processor)
    else:
        data_collator = DataCollatorCTCWithPadding(processor=processor)

    # Define compute metric function
    def compute_metrics(pred):
        pred_logits = pred.predictions
        pred_ids = np.argmax(pred_logits, axis=-1)
        pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id

        pred_strs = processor.batch_decode(pred_ids, skip_special_tokens=True)

        # we do not want to group tokens when computing the metrics
        label_strs = processor.batch_decode(pred.label_ids, group_tokens=False)
        mixed_distance, mixed_tokens = 0, 0
        char_distance, char_tokens = 0, 0

        pred_strs = list(map(lambda pred_str: pred_str[:-1].strip(), pred_strs))
        label_strs = list(map(lambda label_str: label_str.replace('[UNK]','#'), label_strs))
        for pred_str, label_str in zip(pred_strs, label_strs):
            # Calculate
            m_pred = tokenize_for_mer(pred_str)
            m_ref = tokenize_for_mer(label_str)
            mixed_distance += editdistance.distance(m_pred, m_ref)
            mixed_tokens += len(m_ref)

            c_pred = tokenize_for_cer(pred_str)
            c_ref = tokenize_for_cer(label_str)
            char_distance += editdistance.distance(c_pred, c_ref)
            char_tokens += len(c_ref)

        f = open(f'{training_args.output_dir}/test.results', 'w')
        f.writelines([item+'\n' for item in pred_strs])
        f.close()
        f = open(f'{training_args.output_dir}/test.label', 'w')
        f.writelines([item+'\n' for item in label_strs])
        f.close()

        mer = mixed_distance / mixed_tokens
        cer = char_distance / char_tokens

        return {"mer": mer, "cer": cer}

    # Initialize Trainer
    trainer = Trainer(
        model=model,
        data_collator=data_collator,
        args=training_args,
        compute_metrics=compute_metrics,
        train_dataset=vectorized_datasets["test"] if training_args.do_train else None,
        eval_dataset=vectorized_datasets["test"] if training_args.do_eval else None,
        tokenizer=processor.feature_extractor,
    )

    ###
    # Evaluation Phase
    ###
    results = {}
    logger.info("*** Test Phase ***")
    metrics = trainer.evaluate(eval_dataset=vectorized_datasets["test"])
    metrics["eval_samples"] = len(vectorized_datasets["test"])

    trainer.log_metrics("eval", metrics)
    trainer.save_metrics("eval", metrics)

    # Write model card and (optionally) push to hub
    kwargs = {
        "finetuned_from": model_args.model_name_or_path,
        "tasks": "speech-recognition",
        "tags": ["automatic-speech-recognition", "ASCEND"],
        "dataset_args": "Config: na",
        "dataset": "ASCEND",
        "language": "zh-en"
    }

    if training_args.push_to_hub:
        trainer.push_to_hub(**kwargs)
    else:
        trainer.create_model_card(**kwargs)

    return results

#####
# Entry Point
#####
def main():
    ###
    # Parsing & Initialization
    ###
    # Parse argument
    parser = HfArgumentParser((ModelArguments, DataArguments, TrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Set random seed
    set_seed(training_args.seed)

    # Detect last checkpoint
    last_checkpoint = None
    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
        last_checkpoint = get_last_checkpoint(training_args.output_dir)
        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to overcome."
            )
        elif last_checkpoint is not None:
            logger.info(
                f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )

    ###
    # Prepare logger
    ###
    # Init logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    logger.setLevel(logging.INFO if is_main_process(training_args.local_rank) else logging.WARN)

    # Log on each process the small summary:
    logger.warning(
        f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}"
        f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}"
    )
    # Set the verbosity to warn of the Transformers logger (on main process only):
    if is_main_process(training_args.local_rank):
        transformers.utils.logging.set_verbosity(transformers.logging.WARNING)
    logger.info("Training/evaluation parameters %s", training_args)

    ###
    # RUN RUN RUN!!!
    ###
    run(model_args, data_args, training_args)

if __name__ == '__main__':
    main()


text:
from dataclasses import dataclass, field
from typing import Optional
from transformers import TrainingArguments

@dataclass
class ModelArguments:
    """
    Arguments pertaining to which model/config/tokenizer we are going to utilize.
    """
    model_name_or_path: Optional[str] = field(
        default="ydshieh/wav2vec2-large-xlsr-53-chinese-zh-cn-gpt", metadata={"help": "The path of the HuggingFace model."}
    )
    mask_time_prob: float = field(
        default=0.065,
        metadata={
            "help": "Probability of each feature vector along the time axis to be chosen as the start of the vector"
            "span to be masked. Approximately ``mask_time_prob * sequence_length // mask_time_length`` feature"
            "vectors will be masked along the time axis."
        },
    )
    mask_time_length: int = field(
        default=3,
        metadata={"help": "Length of vector span to mask along the time axis."},
    )
    mask_feature_prob: float = field(
        default=0.004,
        metadata={
            "help": "Probability of each feature vector along the feature axis to be chosen as the start of the vector"
            "span to be masked. Approximately ``mask_feature_prob * sequence_length // mask_feature_length`` feature bins will be masked along the time axis."
        },
    )
    mask_feature_length: int = field(
        default=3,
        metadata={"help": "Length of vector span to mask along the feature axis."},
    )

@dataclass
class DataArguments:
    """
    Arguments pertaining to the data loading and preprocessing pipeline.
    """
    train_manifest_path: Optional[str] = field(
        default="dataset/train_metadata.csv", metadata={"help": "The path of the training dataset to use."}
    )
    valid_manifest_path: Optional[str] = field(
        default="dataset/validation_metadata.csv", metadata={"help": "The path of the validation dataset to use."}
    )
    test_manifest_path: Optional[str] = field(
        default="dataset/test_metadata.csv", metadata={"help": "The path of the testing dataset to use."}
    )
    num_workers: Optional[int] = field(
        default=8,
        metadata={"help": "The number of processes to use for the dataset."},
    )
    preprocessing_num_workers: Optional[int] = field(
        default=16,
        metadata={"help": "The number of processes to use for the preprocessing."},
    )
    preprocessing_only: Optional[bool] = field(
        default=False,
        metadata={"help": "Whether to only run preprocessing."},
    )
    audio_column_name: Optional[str] = field(
        default="audio",
        metadata={"help": "The name of the dataset column containing the audio path. Defaults to 'audio'"},
    )
    text_column_name: Optional[str] = field(
        default="text",
        metadata={"help": "The name of the dataset column containing the text data. Defaults to 'text'"},
    )
    video_column_name: Optional[str] = field(
        default=None,
        metadata={"help": "The name of the dataset column containing the video data. Defaults to 'None'"},
    )
    use_video: Optional[bool] = field(
        default=False,
        metadata={"help": "Whether to incorporate video data for ASR or not. Defaults to 'False'"},
    )


text:
import torchaudio
import torch
import re
import pandas as pd
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Union
from transformers import Wav2Vec2Processor
from datasets import Dataset, DatasetDict, load_from_disk, load_dataset, load_metric

#####
# Data Loading Function
#####
def speech_file_to_array_fn(batch):
    speech_array, sampling_rate = torchaudio.load(batch['path'])
    batch["speech_sample"] = speech_array[0].numpy()
    batch["sampling_rate"] = sampling_rate
    return batch

def load_dataset(manifest_file, num_proc, audio_column_name, text_column_name, video_column_name=None):
    base_path = '/'.join(manifest_file.split('/')[:-1])

    manifest_df = pd.read_csv(manifest_file)
    manifest_df['path'] = manifest_df[audio_column_name].apply(lambda path: f'{base_path}/{path}')
    manifest_df['transcription'] = manifest_df[text_column_name].apply(
        lambda path: open(f'{base_path}/{path}', "r").read()
    )
    if video_column_name:
        manifest_df = manifest_df.rename({video_column_name: 'video_path'}, axis='columns')

    batches = Dataset.from_pandas(manifest_df)
    batches = batches.map(speech_file_to_array_fn, num_proc=num_proc)
    return batches


text:
numpy
pandas
torchaudio
torch
librosa
datasets
transformers
jiwer
jieba
editdistance
tensorboard
fairscale


text:
import torch
from torch import nn
import torchvision.models as models

_HIDDEN_STATES_START_POSITION = 2

class MMWav2Vec2Model(nn.Module):
    def __init__(self, wav2vec2ctc):
        super(MMWav2Vec2Model, self).__init__()
        # Wav2Vec2 Model
        self.wav2vec2ctc = wav2vec2ctc

        # Video Model
        self.video_1d_resnet_18 = models.resnet18(pretrained=True)
        self.video_1d_resnet_18.conv1 = nn.Conv2d(in_channels=1, out_channels=64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
        self.video_1d_resnet_18.fc = nn.Identity()
        self.video_time_conv = nn.Conv1d(in_channels=512, out_channels=1024, kernel_size=3, padding='same')
        self.video_lm_head = nn.Linear(1024, self.wav2vec2ctc.config.vocab_size)

    def gradient_checkpointing_enable(self):
        return self.wav2vec2ctc.gradient_checkpointing_enable()

    def encode_audio(self, input_values):
        outputs = self.wav2vec2ctc(input_values)
        return outputs

    def encode_video(self, video_values):
        batch_size = video_values.shape[0]
        seq_len, n_channel = video_values.shape[1], video_values.shape[2]
        width, height = video_values.shape[3], video_values.shape[4]

        video_values = video_values.reshape(-1, n_channel, width, height) # [B, L, C, W, H] => [B*L, C, W, H]
        video_output = self.video_1d_resnet_18(video_values)# [B*L, C, W, H] => [B*L, D]
        video_output = video_output.reshape(batch_size, seq_len, -1) # [B*L, D] => [B, L, D]
        video_output = self.video_time_conv(video_output.transpose(1, 2)) # [B, L, D] => [B, D', L]
        video_logits = self.video_lm_head(video_output.transpose(1, 2)) # [B, D', L] => [B, L, Vocab]

        return video_logits

    def forward(self, input_values, labels=None, attention_mask=None, video_values=None, return_dict=False, *args, **kwargs):
        # encode audio
        outputs = self.encode_audio(input_values, *args, **kwargs)
        logits = outputs.get('logits')

        if video_values is not None:
            # encode video
            video_logits = self.encode_video(video_values, *args, **kwargs)

            # fuse multimodal
            video_logits = video_logits.repeat(1,2,1) # Convert 25 FPS to 50 FPS
            logits = logits + video_logits[:,:logits.shape[1],:] # Fuse with Audio

        if labels is not None:
            if labels.max() >= self.wav2vec2ctc.config.vocab_size:
                raise ValueError(f"Label values must be < vocab_size: {self.wav2vec2ctc.config.vocab_size}")

            # retrieve loss input_lengths from attention_mask
            attention_mask = (
                attention_mask if attention_mask is not None else torch.ones_like(input_values, dtype=torch.long)
            )
            input_lengths = self.wav2vec2ctc._get_feat_extract_output_lengths(attention_mask.sum(-1)).to(torch.long)

            # assuming that padded tokens are filled with -100
            # when not being attended to
            labels_mask = labels >= 0
            target_lengths = labels_mask.sum(-1)
            flattened_targets = labels.masked_select(labels_mask)

            # ctc_loss doesn't support fp16
            log_probs = nn.functional.log_softmax(logits, dim=-1, dtype=torch.float32).transpose(0, 1)

            with torch.backends.cudnn.flags(enabled=False):
                loss = nn.functional.ctc_loss(
                    log_probs,
                    flattened_targets,
                    input_lengths,
                    target_lengths,
                    blank=self.wav2vec2ctc.config.pad_token_id,
                    reduction=self.wav2vec2ctc.config.ctc_loss_reduction,
                    zero_infinity=self.wav2vec2ctc.config.ctc_zero_infinity,
                )

        if not return_dict:
            output = (logits,) + outputs[_HIDDEN_STATES_START_POSITION:]
            return ((loss,) + output) if loss is not None else output

        return CausalLMOutput(
            loss=loss, logits=logits, hidden_states=outputs.hidden_states, attentions=outputs.attentions
        )


text:
CUDA_VISIBLE_DEVICES=0 python eval.py --output_dir=./save_mm_noisy/0/eval \
--model_name_or_path=save_mm/14045/checkpoint-8980  --test_manifest_path=cache_mm_noisy/0/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 --use_video \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=0 python eval.py --output_dir=./save_mm_noisy/1/eval \
--model_name_or_path=save_mm/14045/checkpoint-8980  --test_manifest_path=cache_mm_noisy/1/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 --use_video \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=0 python eval.py --output_dir=./save_mm_noisy/2/eval \
--model_name_or_path=save_mm/14045/checkpoint-8980  --test_manifest_path=cache_mm_noisy/2/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 --use_video \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=0 python eval.py --output_dir=./save_mm_noisy/3/eval \
--model_name_or_path=save_mm/14045/checkpoint-8980  --test_manifest_path=cache_mm_noisy/3/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 --use_video \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=0 python eval.py --output_dir=./save_mm_noisy/4/eval \
--model_name_or_path=save_mm/14045/checkpoint-8980  --test_manifest_path=cache_mm_noisy/4/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 --use_video \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=0 python eval.py --output_dir=./save_mm_noisy/5/eval \
--model_name_or_path=save_mm/14045/checkpoint-8980  --test_manifest_path=cache_mm_noisy/5/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 --use_video \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=0 python eval.py --output_dir=./save_mm_noisy/6/eval \
--model_name_or_path=save_mm/14045/checkpoint-8980  --test_manifest_path=cache_mm_noisy/6/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 --use_video \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=0 python eval.py --output_dir=./save_mm_noisy/7/eval \
--model_name_or_path=save_mm/14045/checkpoint-8980  --test_manifest_path=cache_mm_noisy/7/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 --use_video \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=0 python eval.py --output_dir=./save_mm_noisy/8/eval \
--model_name_or_path=save_mm/14045/checkpoint-8980  --test_manifest_path=cache_mm_noisy/8/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 --use_video \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=0 python eval.py --output_dir=./save_mm_noisy/9/eval \
--model_name_or_path=save_mm/14045/checkpoint-8980  --test_manifest_path=cache_mm_noisy/9/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 --use_video \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True

CUDA_VISIBLE_DEVICES=0 python eval.py --output_dir=./save_mm_noisy/all/eval \
--model_name_or_path=save_mm/14045/checkpoint-8980  --test_manifest_path=cache_mm_noisy/all/preprocess_data.arrow \
--audio_column_name=audio_path --text_column_name=text_path  --video_column_name=lip_image_path \
--per_device_train_batch_size=16 --per_device_eval_batch_size=16     --dataloader_num_workers=32 \
--dataloader_pin_memory     --seed=0 --num_train_epochs=20 --learning_rate=5e-5  \
--logging_strategy=steps --logging_steps=10 \
--evaluation_strategy=epoch --eval_steps=1 --eval_accumulation_steps=30 \
--save_steps=1 --save_strategy=epoch --save_total_limit=1 --use_video \
--metric_for_best_model=mer --greater_is_better=False --load_best_model_at_end=True


text:
import numpy as np
from dataclasses import dataclass
from typing import Union, Optional, List, Dict

import torch
from transformers import Wav2Vec2Processor

#####
# Data Collator CTC
#####
@dataclass
class DataCollatorCTCWithPadding:
    """
    Data collator that will dynamically pad the inputs received.
    Args:
        processor (:class:`~transformers.Wav2Vec2Processor`)
            The processor used for proccessing the data.
        padding (:obj:`bool`, :obj:`str` or :class:`~transformers.tokenization_utils_base.PaddingStrategy`, `optional`, defaults to :obj:`True`):
            Select a strategy to pad the returned sequences (according to the model's padding side and padding index)
            among:
            * :obj:`True` or :obj:`'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
              sequence if provided).
            * :obj:`'max_length'`: Pad to a maximum length specified with the argument :obj:`max_length` or to the
              maximum acceptable input length for the model if that argument is not provided.
            * :obj:`False` or :obj:`'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of
              different lengths).
        max_length (:obj:`int`, `optional`):
            Maximum length of the ``input_values`` of the returned list and optionally padding length (see above).
        max_length_labels (:obj:`int`, `optional`):
            Maximum length of the ``labels`` returned list and optionally padding length (see above).
        pad_to_multiple_of (:obj:`int`, `optional`):
            If set will pad the sequence to a multiple of the provided value.
            This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >=
            7.5 (Volta).
    """

    processor: Wav2Vec2Processor
    padding: Union[bool, str] = True
    max_length: Optional[int] = None
    max_length_labels: Optional[int] = None
    pad_to_multiple_of: Optional[int] = None
    pad_to_multiple_of_labels: Optional[int] = None

    def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
        # split inputs and labels since they have to be of different lenghts and need
        # different padding methods
        input_features = [{"input_values": feature["input_values"]} for feature in features]
        label_features = [{"input_ids": feature["labels"]} for feature in features]

        batch = self.processor.pad(
            input_features,
            padding=self.padding,
            max_length=self.max_length,
            pad_to_multiple_of=self.pad_to_multiple_of,
            return_tensors="pt",
        )
        with self.processor.as_target_processor():
            labels_batch = self.processor.pad(
                label_features,
                padding=self.padding,
                max_length=self.max_length_labels,
                pad_to_multiple_of=self.pad_to_multiple_of_labels,
                return_tensors="pt",
            )

        # replace padding with -100 to ignore loss correctly
        labels = labels_batch["input_ids"].masked_fill(labels_batch.attention_mask.ne(1), -100)
        batch["labels"] = labels

        return batch

#####
# Data Collator MM CTC
#####
@dataclass
class DataCollatorMMCTCWithPadding:
    """
    Data collator that will dynamically pad the inputs received.
    Args:
        processor (:class:`~transformers.Wav2Vec2Processor`)
            The processor used for proccessing the data.
        padding (:obj:`bool`, :obj:`str` or :class:`~transformers.tokenization_utils_base.PaddingStrategy`, `optional`, defaults to :obj:`True`):
            Select a strategy to pad the returned sequences (according to the model's padding side and padding index)
            among:
            * :obj:`True` or :obj:`'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
              sequence if provided).
            * :obj:`'max_length'`: Pad to a maximum length specified with the argument :obj:`max_length` or to the
              maximum acceptable input length for the model if that argument is not provided.
            * :obj:`False` or :obj:`'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of
              different lengths).
        max_length (:obj:`int`, `optional`):
            Maximum length of the ``input_values`` of the returned list and optionally padding length (see above).
        max_length_labels (:obj:`int`, `optional`):
            Maximum length of the ``labels`` returned list and optionally padding length (see above).
        pad_to_multiple_of (:obj:`int`, `optional`):
            If set will pad the sequence to a multiple of the provided value.
            This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >=
            7.5 (Volta).
    """

    processor: Wav2Vec2Processor
    padding: Union[bool, str] = True
    max_length: Optional[int] = None
    max_length_labels: Optional[int] = None
    pad_to_multiple_of: Optional[int] = None
    pad_to_multiple_of_labels: Optional[int] = None

    def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
        # split inputs and labels since they have to be of different lenghts and need
        # different padding methods
        input_features = [{"input_values": feature["input_values"]} for feature in features]
        video_features = [feature["video_values"] for feature in features]
        label_features = [{"input_ids": feature["labels"]} for feature in features]

        batch = self.processor.pad(
            input_features,
            padding=self.padding,
            max_length=self.max_length,
            pad_to_multiple_of=self.pad_to_multiple_of,
            return_tensors="pt",
        )

        video_max_len = max(map(lambda feat: len(feat), video_features))
        channel = len(video_features[0][0])
        width, height = len(video_features[0][0][0]), len(video_features[0][0][0][0])
        video_batch = torch.zeros((len(input_features), video_max_len, channel, width, height))
        for i, feat in enumerate(video_features):
            video_batch[i,:len(feat),...] = torch.FloatTensor(feat)

        with self.processor.as_target_processor():
            labels_batch = self.processor.pad(
                label_features,
                padding=self.padding,
                max_length=self.max_length_labels,
                pad_to_multiple_of=self.pad_to_multiple_of_labels,
                return_tensors="pt",
            )

        # replace padding with -100 to ignore loss correctly
        labels = labels_batch["input_ids"].masked_fill(labels_batch.attention_mask.ne(1), -100)

        batch["video_values"] = video_batch
        batch["labels"] = labels

        return batch


text:
import os, sys
import logging
import numpy as np
import pandas as pd
import argparse
import glob

import torchaudio
import torch
import re
import json
import librosa
from datasets import DatasetDict
import torchvision.transforms as T
import torchvision

from transformers import (
    set_seed,
    Wav2Vec2Processor,
    Wav2Vec2CTCTokenizer,
    Wav2Vec2FeatureExtractor,
    Wav2Vec2ForCTC,
    Wav2Vec2Config,
    Trainer,
    TrainingArguments,
    HfArgumentParser,
    EarlyStoppingCallback
)

from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Union
import datasets
import pickle

import editdistance
import jieba
from itertools import chain

import transformers
from transformers.trainer_utils import get_last_checkpoint, is_main_process
from transformers.utils import check_min_version
from transformers.utils.versions import require_version

from args_helper import ModelArguments, DataArguments

import datasets
from datasets import load_from_disk, set_caching_enabled

from utils import CHARS_TO_IGNORE, remove_special_characters, extract_all_chars, tokenize_for_mer, tokenize_for_cer
from data_utils import speech_file_to_array_fn, load_dataset
from data_collator_ctc import DataCollatorCTCWithPadding, DataCollatorMMCTCWithPadding
from mm_wrapper import MMWav2Vec2Model

set_caching_enabled(True)
logger = logging.getLogger(__name__)

#####
# Main Functions
#####
def run(model_args, data_args, training_args):
    ###
    # Prepare Processor & Model
    ###
    training_args.gradient_checkpointing = True
    print('Load Wav2Vec2 processor...')
    processor = Wav2Vec2Processor.from_pretrained(model_args.model_name_or_path)
    base_path = '/'.join(data_args.test_manifest_path.split('/')[:-1])

    ###
    # Prepare Dataset
    ###
    raw_datasets = DatasetDict()
    print('Loading test dataset...')
    raw_datasets["test"] = load_dataset(data_args.test_manifest_path, data_args.preprocessing_num_workers,
            data_args.audio_column_name, data_args.text_column_name, data_args.video_column_name)

    print('Preprocess dataset...')

    # Remove ignorable characters
    print('Removing ignorable characters')
    chars_to_ignore_re = f"[{re.escape(''.join(CHARS_TO_IGNORE))}]"
    def remove_special_characters(batch):
        if chars_to_ignore_re is not None:
            batch['transcription'] = re.sub(chars_to_ignore_re, "", batch['transcription']).lower() + " "
        else:
            batch['transcription'] = batch['transcription'].lower() + " "
        return batch

    with training_args.main_process_first(desc="dataset map special characters removal"):
        raw_datasets = raw_datasets.map(
            remove_special_characters,
            num_proc=data_args.preprocessing_num_workers,
            desc="remove special characters from datasets",
            load_from_cache_file=True
        )

    # Preprocess audio sample and label text
    print('Vectorize dataset...')

    def prepare_dataset(batch):
        # Preprocess audio
        batch["input_values"] = processor(batch["speech_sample"]).input_values[0]

        # Preprocess text
        with processor.as_target_processor():
            batch["labels"] = processor(batch['transcription']).input_ids

        return batch

    removable_column_names = raw_datasets["test"].column_names
    if data_args.use_video:
        removable_column_names.remove('video_path')

    with training_args.main_process_first(desc="dataset map preprocessing"):
        vectorized_datasets = raw_datasets.map(
            prepare_dataset,
            remove_columns=removable_column_names,
            num_proc=data_args.preprocessing_num_workers,
            desc="preprocess datasets",
            load_from_cache_file=False
        )

    # Preprocess video sample
    if data_args.use_video:
        print('Load video data...')
        img_transforms = T.Compose([
            T.Grayscale(num_output_channels=1),
            T.Resize((32,32))
        ])

        def load_video_data(batch):
            image_buffers = []
            video_path = batch["video_path"]
            for image_path in glob.glob(f'{base_path}/{video_path}/*.jpg'):
                image = torchvision.io.read_image(image_path) / 255
                image = img_transforms(image)
                image_buffers.append(image)
            batch["video_values"] = image_buffers # L, C, H, W
            return batch

        with training_args.main_process_first(desc="dataset map preprocessing"):
            vectorized_datasets = vectorized_datasets.map(
                load_video_data,
                remove_columns=['video_path'],
                num_proc=data_args.preprocessing_num_workers,
                desc="preprocess datasets",
                load_from_cache_file=False
            )
    vectorized_datasets.save_to_disk(f'{training_args.output_dir}/preprocess_data.arrow')

    logger.info(f"Data preprocessing finished. Files cached at {training_args.output_dir}/preprocess_data.arrow")
    return

#####
# Entry Point
#####
def main():
    ###
    # Parsing & Initialization
    ###
    # Parse argument
    parser = HfArgumentParser((ModelArguments, DataArguments, TrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Set random seed
    set_seed(training_args.seed)

    ###
    # Prepare logger
    ###
    # Init logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    logger.setLevel(logging.INFO if is_main_process(training_args.local_rank) else logging.WARN)

    # Log on each process the small summary:
    logger.warning(
        f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}"
        f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}"
    )
    # Set the verbosity to warn of the Transformers logger (on main process only):
    if is_main_process(training_args.local_rank):
        transformers.utils.logging.set_verbosity(transformers.logging.WARNING)
    logger.info("Training/evaluation parameters %s", training_args)

    ###
    # RUN RUN RUN!!!
    ###
    run(model_args, data_args, training_args)

if __name__ == '__main__':
    main()


text:
# CI-AVSR
Repository for the paper [CI-AVSR: A Cantonese Audio-Visual Speech Dataset for In-car Command Recognition](https://arxiv.org/pdf/2201.03804.pdf) and the corresponding new dataset, which is accepted in LREC 2022.

If you find our dataset or code useful, please cite this paper, thanks!
<pre>
@article{Dai2022CIAVSRAC,
  title={CI-AVSR: A Cantonese Audio-Visual Speech Dataset for In-car Command Recognition},
  author={Wenliang Dai and Samuel Cahyawijaya and Tiezheng Yu and Elham J. Barezi and Peng Xu and Cheuk Tung Shadow Yiu and Rita Frieske and Holy Lovenia and Genta Indra Winata and Qifeng Chen and Xiaojuan Ma and Bertram E. Shi and Pascale Fung},
  journal={ArXiv},
  year={2022},
  volume={abs/2201.03804}
}
</pre>

# Data
***Version 1.0***

For details of the dataset, please refer to the paper.

### Clean Sets
The originally collected data, including `train_clean.csv`, `valid_clean.csv`, `test_clean.csv`.

### Noise Augmented Sets
The noise augmented data, including `train_noisy.csv`, `valid_noisy.csv`, `test_noisy.csv`. In addition, there is also a out-of-domain test set `test_noisy_ood.csv` to evaluate the generalization of models, i.e. these noises are not in the training set.

### Data Download
[[CI-AVSR Dataset]](https://drive.google.com/file/d/1ov4hfcL4oPncE6H4ZeAwJRp9dpUJP1Zr/view?usp=sharing) We provide the processed CI-AVSR dataset with audios, image frames (25 per seconds), annotations, and augmentation.

[[Raw Videos]](https://drive.google.com/file/d/1uv-IUGx0Msxo1WHZxKGqYMwXHTulUyl4/view?usp=sharing) Just in case, we also provide the unprocessed raw videos (you may not need to use them).


text:
import os, sys
import logging
import numpy as np
import pandas as pd
import argparse
import glob

import torchaudio
import torch
import re
import json
import librosa
from datasets import DatasetDict
import torchvision.transforms as T
import torchvision

from transformers import (
    set_seed,
    Wav2Vec2Processor,
    Wav2Vec2CTCTokenizer,
    Wav2Vec2FeatureExtractor,
    Wav2Vec2ForCTC,
    Wav2Vec2Config,
    Trainer,
    TrainingArguments,
    HfArgumentParser,
    EarlyStoppingCallback
)

from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional, Union
import datasets
import pickle

import editdistance
import jieba
from itertools import chain

import transformers
from transformers.trainer_utils import get_last_checkpoint, is_main_process
from transformers.utils import check_min_version
from transformers.utils.versions import require_version

from args_helper import ModelArguments, DataArguments

import datasets
from datasets import load_from_disk, set_caching_enabled

from utils import CHARS_TO_IGNORE, remove_special_characters, extract_all_chars, tokenize_for_mer, tokenize_for_cer
from data_utils import speech_file_to_array_fn, load_dataset
from data_collator_ctc import DataCollatorCTCWithPadding, DataCollatorMMCTCWithPadding
from mm_wrapper import MMWav2Vec2Model

set_caching_enabled(True)
logger = logging.getLogger(__name__)

#####
# Main Functions
#####
def run(model_args, data_args, training_args):
    ###
    # Prepare Processor & Model
    ###
    training_args.gradient_checkpointing = True
    print('Load Wav2Vec2 model and processor...')
    config = Wav2Vec2Config.from_pretrained(model_args.model_name_or_path)
    config.update({
        "mask_time_prob": model_args.mask_time_prob,
        "mask_time_length": model_args.mask_time_length,
        "mask_feature_prob": model_args.mask_feature_prob,
        "mask_feature_length": model_args.mask_feature_length,
        "gradient_checkpointing": training_args.gradient_checkpointing,
    })
    processor = Wav2Vec2Processor.from_pretrained(model_args.model_name_or_path)
    wav2vec2ctc = Wav2Vec2ForCTC.from_pretrained(model_args.model_name_or_path, config=config)
    if data_args.use_video:
        model = MMWav2Vec2Model(wav2vec2ctc)
    else:
        model = wav2vec2ctc
    model.cuda()

    if data_args.use_video:
        cache_file = './cache_mm/preprocess_data.arrow'
        cache_folder = './cache_mm'
    else:
        cache_file = './cache/preprocess_data.arrow'
        cache_folder = './cache'

    if not os.path.exists(cache_file):
        base_path = '/'.join(data_args.train_manifest_path.split('/')[:-1])

        ###
        # Prepare Dataset
        ###
        raw_datasets = DatasetDict()
        print('Loading train dataset...')
        raw_datasets["train"] = load_dataset(data_args.train_manifest_path, data_args.num_workers,
                data_args.audio_column_name, data_args.text_column_name, data_args.video_column_name)
        print('Loading validation dataset...')
        raw_datasets["valid"] = load_dataset(data_args.valid_manifest_path, data_args.num_workers,
                data_args.audio_column_name, data_args.text_column_name, data_args.video_column_name)
        print('Loading test dataset...')
        raw_datasets["test"] = load_dataset(data_args.test_manifest_path, data_args.num_workers,
                data_args.audio_column_name, data_args.text_column_name, data_args.video_column_name)

        print('Preprocess dataset...')

        # Remove ignorable characters
        print('Removing ignorable characters')
        chars_to_ignore_re = f"[{re.escape(''.join(CHARS_TO_IGNORE))}]"
        def remove_special_characters(batch):
            if chars_to_ignore_re is not None:
                batch['transcription'] = re.sub(chars_to_ignore_re, "", batch['transcription']).lower() + " "
            else:
                batch['transcription'] = batch['transcription'].lower() + " "
            return batch

        with training_args.main_process_first(desc="dataset map special characters removal"):
            raw_datasets = raw_datasets.map(
                remove_special_characters,
                num_proc=data_args.preprocessing_num_workers,
                desc="remove special characters from datasets",
                load_from_cache_file=True,
                cache_file_names={
                    "train": f"{cache_folder}/train_clean.arrow",
                    "valid": f"{cache_folder}/valid_clean.arrow",
                    "test": f"{cache_folder}/test_clean.arrow"
                }
            )

        # Preprocess audio sample and label text
        print('Vectorize dataset...')

        def prepare_dataset(batch):
            # Preprocess audio
            batch["input_values"] = processor(batch["speech_sample"]).input_values[0]

            # Preprocess text
            with processor.as_target_processor():
                batch["labels"] = processor(batch['transcription']).input_ids

            return batch

        removable_column_names = raw_datasets["train"].column_names
        if data_args.use_video:
            removable_column_names.remove('video_path')

        with training_args.main_process_first(desc="dataset map preprocessing"):
            vectorized_datasets = raw_datasets.map(
                prepare_dataset,
                remove_columns=removable_column_names,
                num_proc=data_args.preprocessing_num_workers,
                desc="preprocess datasets",
                load_from_cache_file=False,
                cache_file_names={
                    "train": f"{cache_folder}/train_vec.arrow",
                    "valid": f"{cache_folder}/valid_vec.arrow",
                    "test": f"{cache_folder}/test_vec.arrow"
                }
            )

        # Preprocess video sample
        if data_args.use_video:
            print('Load video data...')
            img_transforms = T.Compose([
                T.Grayscale(num_output_channels=1),
                T.Resize((32,32))
            ])

            def load_video_data(batch):
                image_buffers = []
                video_path = batch["video_path"]
                for image_path in glob.glob(f'{base_path}/{video_path}/*.jpg'):
                    image = torchvision.io.read_image(image_path) / 255
                    image = img_transforms(image)
                    image_buffers.append(image)
                batch["video_values"] = image_buffers # L, C, H, W
                return batch

            with training_args.main_process_first(desc="dataset map preprocessing"):
                vectorized_datasets = vectorized_datasets.map(
                    load_video_data,
                    remove_columns=['video_path'],
                    num_proc=data_args.preprocessing_num_workers,
                    desc="preprocess datasets",
                    load_from_cache_file=False,
                    cache_file_names={
                        "train": f"{cache_folder}/train_vec.arrow",
                        "valid": f"{cache_folder}/valid_vec.arrow",
                        "test": f"{cache_folder}/test_vec.arrow"
                    }
                )

        vectorized_datasets.save_to_disk(f'{cache_folder}/preprocess_data.arrow')
    else:
        print('Loading cached dataset...')
        vectorized_datasets = datasets.load_from_disk(f'{cache_folder}/preprocess_data.arrow')

    if data_args.preprocessing_only:
        logger.info(f"Data preprocessing finished. Files cached at {vectorized_datasets.cache_files}")
        return

    ###
    # Prepare Data Collator and Trainer
    ###
    print('Preparing Trainer...')

    # Instantiate custom data collator
    if data_args.use_video:
        data_collator = DataCollatorMMCTCWithPadding(processor=processor)
    else:
        data_collator = DataCollatorCTCWithPadding(processor=processor)

    # Define compute metric function
    def compute_metrics(pred):
        pred_logits = pred.predictions
        pred_ids = np.argmax(pred_logits, axis=-1)
        pred.label_ids[pred.label_ids == -100] = processor.tokenizer.pad_token_id

        pred_strs = processor.batch_decode(pred_ids)

        # we do not want to group tokens when computing the metrics
        label_strs = processor.batch_decode(pred.label_ids, group_tokens=False)
        mixed_distance, mixed_tokens = 0, 0
        char_distance, char_tokens = 0, 0

        pred_strs = list(map(lambda pred_str: pred_str[:-1].strip(), pred_strs))
        label_strs = list(map(lambda label_str: label_str.replace('[UNK]','#'), label_strs))
        for pred_str, label_str in zip(pred_strs, label_strs):
            # Calculate
            m_pred = tokenize_for_mer(pred_str)
            m_ref = tokenize_for_mer(label_str)
            mixed_distance += editdistance.distance(m_pred, m_ref)
            mixed_tokens += len(m_ref)

            c_pred = tokenize_for_cer(pred_str)
            c_ref = tokenize_for_cer(label_str)
            char_distance += editdistance.distance(c_pred, c_ref)
            char_tokens += len(c_ref)
        mer = mixed_distance / mixed_tokens
        cer = char_distance / char_tokens

        f = open(f'{training_args.output_dir}/valid.results', 'w')
        f.writelines([item+'\n' for item in pred_strs])
        f.close()
        f = open(f'{training_args.output_dir}/valid.label', 'w')
        f.writelines([item+'\n' for item in label_strs])
        f.close()

        return {"mer": mer, "cer": cer}

    # Initialize Trainer
    trainer = Trainer(
        train_dataset=vectorized_datasets["train"],
        eval_dataset=vectorized_datasets["valid"],
        model=model,
        data_collator=data_collator,
        args=training_args,
        compute_metrics=compute_metrics,
        tokenizer=processor.feature_extractor,
        callbacks = [EarlyStoppingCallback(early_stopping_patience=5)]
    )

    ###
    # Training Phase
    ###
    print('*** Training Phase ***')

    # use last checkpoint if exist
    if os.path.isdir(model_args.model_name_or_path):
        checkpoint = model_args.model_name_or_path
    else:
        checkpoint = None

    train_result = trainer.train(resume_from_checkpoint=checkpoint)
    trainer.save_model()

    # Save the feature_extractor and the tokenizer
    if is_main_process(training_args.local_rank):
        processor.save_pretrained(training_args.output_dir)

    metrics = train_result.metrics
    metrics["train_samples"] = len(vectorized_datasets["train"])

    trainer.log_metrics("train", metrics)
    trainer.save_metrics("train", metrics)
    trainer.save_state()

    ###
    # Evaluation Phase
    ###
    results = {}
    logger.info("*** Evaluation Phase ***")
    metrics = trainer.evaluate(eval_dataset=vectorized_datasets["valid"])
    metrics["eval_samples"] = len(vectorized_datasets["valid"])

    trainer.log_metrics("eval", metrics)
    trainer.save_metrics("eval", metrics)

    pickle.dump(metrics, open(f'{training_args.output_dir}/results.pkl', 'wb'))

    print('=== Valid Performance ===')
    for k, v in metrics.items():
        print('{:>30}: {:<50}'.format(k, str(v)).center(80))

#####
# Entry Point
#####
def main():
    ###
    # Parsing & Initialization
    ###
    # Parse argument
    parser = HfArgumentParser((ModelArguments, DataArguments, TrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Set random seed
    set_seed(training_args.seed)

    # Detect last checkpoint
    last_checkpoint = None
    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
        last_checkpoint = get_last_checkpoint(training_args.output_dir)
        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to overcome."
            )
        elif last_checkpoint is not None:
            logger.info(
                f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )

    ###
    # Prepare logger
    ###
    # Init logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    logger.setLevel(logging.INFO if is_main_process(training_args.local_rank) else logging.WARN)

    # Log on each process the small summary:
    logger.warning(
        f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}"
        f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}"
    )
    # Set the verbosity to warn of the Transformers logger (on main process only):
    if is_main_process(training_args.local_rank):
        transformers.utils.logging.set_verbosity(transformers.logging.WARNING)
    logger.info("Training/evaluation parameters %s", training_args)

    ###
    # RUN RUN RUN!!!
    ###
    run(model_args, data_args, training_args)

if __name__ == '__main__':
    main()