Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypotheses

TL;DR

DualHyp tackles brittleness in audio-visual speech recognition caused by modality-specific corruption by maintaining independent ASR and VSR hypothesis streams and performing compositional reasoning in the language space. It introduces RelPrompt, which injects time-aligned reliability signals—$\mathbf{m}^{\text{a}}$ and $\mathbf{m}^{\text{v}}$—to guide the LLM's blending of dual hypotheses, implemented via a reliability mask predictor that outputs tokens $\{\text{Clean}, \text{Noisy}, \text{Mixed}\}$. The approach yields large gains on the LRS2 benchmark under diverse corruptions, with relative WER reductions up to $57.7\%$ over a strong ASR baseline, and demonstrates scalability with larger LLMs and multilingual AVSR settings; a dataset of DualHyp hypotheses is released to support further research. Overall, the work shows that late, language-space fusion of modality-specific evidences, guided by explicit reliability signals, provides a flexible and effective paradigm for robust AVSR in noisy real-world conditions.

Abstract

This paper introduces a new paradigm for generative error correction (GER) framework in audio-visual speech recognition (AVSR) that reasons over modality-specific evidences directly in the language space. Our framework, DualHyp, empowers a large language model (LLM) to compose independent N-best hypotheses from separate automatic speech recognition (ASR) and visual speech recognition (VSR) models. To maximize the effectiveness of DualHyp, we further introduce RelPrompt, a noise-aware guidance mechanism that provides modality-grounded prompts to the LLM. RelPrompt offers the temporal reliability of each modality stream, guiding the model to dynamically switch its focus between ASR and VSR hypotheses for an accurate correction. Under various corruption scenarios, our framework attains up to 57.7% error rate gain on the LRS2 benchmark over standard ASR baseline, contrary to single-stream GER approaches that achieve only 10% gain. To facilitate research within our DualHyp framework, we release the code and the dataset comprising ASR and VSR hypotheses at https://github.com/sungnyun/dualhyp.

Figures (5)

• Figure 1: (a) Conventional GER frameworks use a single set of ASR hypotheses and (optionally) injects visual features via an adapter or a multimodal encoder. (b) Our DualHyp framework maintains modality separation, using both ASR and VSR heads to generate two distinct sets of textual hypotheses. The LLM performs compositional reasoning on dual hypotheses in the language space to produce a more robust and accurate transcription.
• Figure 2: An overview of our DualHyp with RelPrompt. Each predictor uses ASR/VSR encoder features to generate a noise-aware token sequence. These masks accurately guide the LLM to dynamically switch the model's focus between the ASR and VSR hypotheses.
• Figure 3: Qualitative analysis comparing RelPrompt to the DualHyp baseline. RelPrompt uses reliability tokens (i.e., masks) to explicitly inform the input signal quality, correctly guiding the use of ASR and VSR hypotheses.
• Figure 4: WERR at different audio SNRs, under speech noise. Higher WERR indicates greater improvement over the Whisper ASR baseline.
• Figure 5: Word error rate reduction (WERR) at different audio SNRs, under diverse types of noise. Higher WERR indicates greater improvement over the Whisper ASR baseline. The experimental setup is identical to Table \ref{['tab:main_lrs2']}a.