Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognition

Sreyan Ghosh, Sonal Kumar, Ashish Seth, Purva Chiniya, Utkarsh Tyagi, Ramani Duraiswami, Dinesh Manocha

TL;DR

LipGER addresses robust ASR in noisy environments by reframing audio-visual speech recognition as a visually-conditioned generative error correction task. It uses an LLM, prompted with an $N$-best hypothesis list, and conditioned on lip-motion cues through a multi-modal adapter, to rewrite the transcription. The authors also introduce LipHyp, a large-scale dataset of hypothesis-transcription pairs with lip cues, and demonstrate that LipGER yields notable Word Error Rate reductions (up to $49.2\%$ in some settings) across four datasets, including challenging real-world and domain-shift scenarios. This language-space denoising approach reduces the reliance on large-scale paired audio-visual data and offers robust performance across languages, accents, and recording conditions, with practical impact for real-world noisy ASR deployments.

Abstract

Visual cues, like lip motion, have been shown to improve the performance of Automatic Speech Recognition (ASR) systems in noisy environments. We propose LipGER (Lip Motion aided Generative Error Correction), a novel framework for leveraging visual cues for noise-robust ASR. Instead of learning the cross-modal correlation between the audio and visual modalities, we make an LLM learn the task of visually-conditioned (generative) ASR error correction. Specifically, we instruct an LLM to predict the transcription from the N-best hypotheses generated using ASR beam-search. This is further conditioned on lip motions. This approach addresses key challenges in traditional AVSR learning, such as the lack of large-scale paired datasets and difficulties in adapting to new domains. We experiment on 4 datasets in various settings and show that LipGER improves the Word Error Rate in the range of 1.1%-49.2%. We also release LipHyp, a large-scale dataset with hypothesis-transcription pairs that is additionally equipped with lip motion cues to promote further research in this space

LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognition

TL;DR

LipGER addresses robust ASR in noisy environments by reframing audio-visual speech recognition as a visually-conditioned generative error correction task. It uses an LLM, prompted with an -best hypothesis list, and conditioned on lip-motion cues through a multi-modal adapter, to rewrite the transcription. The authors also introduce LipHyp, a large-scale dataset of hypothesis-transcription pairs with lip cues, and demonstrate that LipGER yields notable Word Error Rate reductions (up to in some settings) across four datasets, including challenging real-world and domain-shift scenarios. This language-space denoising approach reduces the reliance on large-scale paired audio-visual data and offers robust performance across languages, accents, and recording conditions, with practical impact for real-world noisy ASR deployments.

Abstract

Visual cues, like lip motion, have been shown to improve the performance of Automatic Speech Recognition (ASR) systems in noisy environments. We propose LipGER (Lip Motion aided Generative Error Correction), a novel framework for leveraging visual cues for noise-robust ASR. Instead of learning the cross-modal correlation between the audio and visual modalities, we make an LLM learn the task of visually-conditioned (generative) ASR error correction. Specifically, we instruct an LLM to predict the transcription from the N-best hypotheses generated using ASR beam-search. This is further conditioned on lip motions. This approach addresses key challenges in traditional AVSR learning, such as the lack of large-scale paired datasets and difficulties in adapting to new domains. We experiment on 4 datasets in various settings and show that LipGER improves the Word Error Rate in the range of 1.1%-49.2%. We also release LipHyp, a large-scale dataset with hypothesis-transcription pairs that is additionally equipped with lip motion cues to promote further research in this space
Paper Structure (13 sections, 3 equations, 3 figures, 1 table)

This paper contains 13 sections, 3 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. LipHyp
  5. ASR Models
  6. Datasets
  7. LipGER
  8. Lip Motion Encoder
  9. Lip Motion Conditioned LLM
  10. Experiments and Results
  11. Experimental Setup
  12. Results
  13. Conclusion and Limitations

Figures (3)

  • Figure 1: Comparison of traditional AVSR methods and LipGER. LipGER benefits over these methods on several aspects by overcoming the requirement of E2E traning.
  • Figure 2: Illustration of LipGer. LipGER performs generative error correction on the $N$-best hypotheses generated by beam search decoding from an ASR model. LipGER is built on an LLM, which performs multi-modal reasoning by conditioning on lip motions. Specifically, we build a prompt from the (N+1)-best hypotheses list, which instructs the LLM to rewrite the best hypothesis using the other hypotheses. The lip motion is conditioned on the LLM using multi-modal adapters with encodings obtained from a lip encoder $L$, and only the adapters and $L$ are trained during the fine-tuning stage.
  • Figure 3: Illustration of predictions by LipGER. Instances I. and II. show positive cases where LipGER predicts the correct transcription by revising the best hypothesis with accurate tokens hidden in the other $N$-best hypotheses. Case III. shows a failure case highlighting the limitation of LipGER, which fails to generate the correct transcription due to a lack of sufficient contextual cues in the other $N$-best hypotheses.