Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representation

TL;DR

The paper tackles the high computational barrier of sentence-level multilingual visual speech recognition by introducing discretized visual speech units derived from a multilingual AV-HuBERT model. It pre-trains a unit-to-text Transformer on discrete inputs/outputs and uses curriculum learning to progressively shift from audio-visual to visual-only inputs, achieving substantial data-efficiency and speed-ups. Finetuning on continuous features enables the model to reach state-of-the-art multilingual VSR performance with a single trained model, competitive with language-specific approaches across Es, It, Fr, and Pt, while maintaining solid English results. The method offers practical benefits for scalable multilingual lip-reading, including drastically reduced data size, faster training, and strong cross-language performance. The work also provides insights into the linguistic content of visual speech units and the degree to which they suppress non-linguistic information such as speaker identity.

Abstract

This paper explores sentence-level multilingual Visual Speech Recognition (VSR) that can recognize different languages with a single trained model. As the massive multilingual modeling of visual data requires huge computational costs, we propose a novel training strategy, processing with visual speech units. Motivated by the recent success of the audio speech unit, we propose to use a visual speech unit that can be obtained by discretizing the visual speech features extracted from the self-supervised visual speech model. Through analysis, we verify that the visual speech units mainly contain viseme information while suppressing non-linguistic information. By using the visual speech units as the inputs of our system, we propose to pre-train a VSR model to predict corresponding text outputs on multilingual data constructed by merging several VSR databases. As both the inputs (i.e., visual speech units) and outputs (i.e., text) are discrete, we can greatly improve the training efficiency compared to the standard VSR training. Specifically, the input data size is reduced to 0.016% of the original video inputs. In order to complement the insufficient visual information in speech recognition, we apply curriculum learning where the inputs of the system begin with audio-visual speech units and gradually change to visual speech units. After pre-training, the model is finetuned on continuous features. We set new state-of-the-art multilingual VSR performances by achieving comparable performances to the previous language-specific VSR models, with a single trained model.

Figures (6)

• Figure 1: Illustration of the proposed multilingual VSR framework. (a) Visual speech units are obtained by quantizing the visual speech features extracted from mAV-HuBERT. (b) The Transformer encoder-decoder model is pre-trained with discrete inputs and outputs. The task difficulty is gradually increased by using progressive masking $\mathcal{M}$. Pre-training commences with audio-visual speech units as inputs, and these inputs gradually transition to visual speech units through progressive masking of the audio speech units. (c) After pre-training the model with discrete inputs and outputs, it is finetuned with continuous features to boost the VSR performances.
• Figure 2: Finetuning efficiency comparison between the proposed pre-training scheme and without the pre-training.
• Figure 3: Visualization of speech units. Each boundary represents a single unit and the same color represents the same phoneme or phoneme family. (a) Audio speech unit. (b) Visual speech unit.
• Figure 4: Example frames corresponding to each visual speech unit. The numbers indicate the classes of visual speech units.
• Figure 5: Visualization of speech units. Each boundary represents a single unit and the same color represents the same phoneme or phoneme family. (a) Audio speech unit. (b) Visual speech unit.