Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, Abdelrahman Mohamed

TL;DR

This paper introduces AV-HuBERT, a self-supervised framework that learns audio-visual speech representations by masking multi-stream inputs and predicting iteratively refined multimodal cluster assignments. By integrating both audio and lip movements, AV-HuBERT achieves state-of-the-art lip-reading with only 30 hours of labeled data and strong ASR performance (1.3% WER) without external language models. The approach relies on modality dropout, multimodal clustering, and a novel imposter-based masking strategy, with extensive ablations showing the benefits of cross-modal targets and iterative refinement. The work demonstrates significant practical impact for low-resource languages and broad speech-related tasks, and provides code and models for reproducibility.

Abstract

Video recordings of speech contain correlated audio and visual information, providing a strong signal for speech representation learning from the speaker's lip movements and the produced sound. We introduce Audio-Visual Hidden Unit BERT (AV-HuBERT), a self-supervised representation learning framework for audio-visual speech, which masks multi-stream video input and predicts automatically discovered and iteratively refined multimodal hidden units. AV-HuBERT learns powerful audio-visual speech representation benefiting both lip-reading and automatic speech recognition. On the largest public lip-reading benchmark LRS3 (433 hours), AV-HuBERT achieves 32.5% WER with only 30 hours of labeled data, outperforming the former state-of-the-art approach (33.6%) trained with a thousand times more transcribed video data (31K hours). The lip-reading WER is further reduced to 26.9% when using all 433 hours of labeled data from LRS3 and combined with self-training. Using our audio-visual representation on the same benchmark for audio-only speech recognition leads to a 40% relative WER reduction over the state-of-the-art performance (1.3% vs 2.3%). Our code and models are available at https://github.com/facebookresearch/av_hubert

Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

TL;DR

This paper introduces AV-HuBERT, a self-supervised framework that learns audio-visual speech representations by masking multi-stream inputs and predicting iteratively refined multimodal cluster assignments. By integrating both audio and lip movements, AV-HuBERT achieves state-of-the-art lip-reading with only 30 hours of labeled data and strong ASR performance (1.3% WER) without external language models. The approach relies on modality dropout, multimodal clustering, and a novel imposter-based masking strategy, with extensive ablations showing the benefits of cross-modal targets and iterative refinement. The work demonstrates significant practical impact for low-resource languages and broad speech-related tasks, and provides code and models for reproducibility.

Abstract

Video recordings of speech contain correlated audio and visual information, providing a strong signal for speech representation learning from the speaker's lip movements and the produced sound. We introduce Audio-Visual Hidden Unit BERT (AV-HuBERT), a self-supervised representation learning framework for audio-visual speech, which masks multi-stream video input and predicts automatically discovered and iteratively refined multimodal hidden units. AV-HuBERT learns powerful audio-visual speech representation benefiting both lip-reading and automatic speech recognition. On the largest public lip-reading benchmark LRS3 (433 hours), AV-HuBERT achieves 32.5% WER with only 30 hours of labeled data, outperforming the former state-of-the-art approach (33.6%) trained with a thousand times more transcribed video data (31K hours). The lip-reading WER is further reduced to 26.9% when using all 433 hours of labeled data from LRS3 and combined with self-training. Using our audio-visual representation on the same benchmark for audio-only speech recognition leads to a 40% relative WER reduction over the state-of-the-art performance (1.3% vs 2.3%). Our code and models are available at https://github.com/facebookresearch/av_hubert
Paper Structure (32 sections, 4 equations, 5 figures, 11 tables)

This paper contains 32 sections, 4 equations, 5 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Preliminary: Audio HuBERT
  5. Single-Modal & Cross-Modal Visual HuBERT
  6. Audio-visual HuBERT
  7. Fine-tuning
  8. Experiment
  9. Setup
  10. Main Result
  11. AV-HuBERT vs. Visual HuBERT
  12. Multilingual vs. Monolingual
  13. ASR performance
  14. Conclusion
  15. Model Illustration
  16. ...and 17 more sections

Figures (5)

  • Figure 1: Illustration of AV-HuBERT. Masked prediction losses are only computed for the three middle frames, because at least one modality is masked for those frames. See section \ref{['sec:app-all-model-figure']} for its comparison between single-modal and cross-modal visual HuBERT.
  • Figure A.1: Comparison between the proposed AV-HuBERT with single-modal and cross-modal visual HuBERT
  • Figure E.1: Quality of feature clusters from different layers across different iterations (BASE, 433 hours unlabeled data). (Iter $i$, Layer $j$): cluster quality of layer-$j$ feature of iter-$i$ model. Upper row: 100, 500, 1K, 2K clusters for 4 iterations. Bottom row: 2K clusters for all iterations. Purity/NMI of the initial MFCC clusters: 30.3%/21.5%
  • Figure F.1: WER vs. sentence length for lip reading (left) and ASR (right)
  • Figure F.2: Transcriptions from different lip-reading models. GT: ground-truth, Proposed: self-supervised model, Supervised: supervised model. Red: wrong words in the output