A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition

TL;DR

The paper tackles the paradoxical effect of dropout on video in Audio-Visual Speech Recognition, where dropout improves robustness to missing video yet degrades performance on fully observed inputs. It introduces the Modality Bias Hypothesis (MBH) and the Modality Bias Diagram (MBVD) to explain how training-time video dropout shifts the latent decision space toward an audio-dominant, unimodal distribution, reducing reliance on visual cues. To address this, the authors propose Multimodal Distribution Approximation with Knowledge Distillation (MDA-KD), which anchors a student model on the teacher’s unimodal-agnostic distributions while learning under missing video, and a Modality-Specific Adapter (MS-Adapter) that dynamically switches to audio-dominant decision when video is severely missing. Experimental validation on MISP2021/2022 shows that MDA-KD plus MS-Adapter achieve strong robustness without sacrificing complete-input performance, outperforming other dropout techniques and attaining competitive SOTA results. The work provides practical mechanisms for robust AVSR in real-world, imperfect video conditions and suggests broader applicability of MBH/MBVD-inspired strategies to multimodal systems.

Abstract

Advanced Audio-Visual Speech Recognition (AVSR) systems have been observed to be sensitive to missing video frames, performing even worse than single-modality models. While applying the dropout technique to the video modality enhances robustness to missing frames, it simultaneously results in a performance loss when dealing with complete data input. In this paper, we investigate this contrasting phenomenon from the perspective of modality bias and reveal that an excessive modality bias on the audio caused by dropout is the underlying reason. Moreover, we present the Modality Bias Hypothesis (MBH) to systematically describe the relationship between modality bias and robustness against missing modality in multimodal systems. Building on these findings, we propose a novel Multimodal Distribution Approximation with Knowledge Distillation (MDA-KD) framework to reduce over-reliance on the audio modality and to maintain performance and robustness simultaneously. Finally, to address an entirely missing modality, we adopt adapters to dynamically switch decision strategies. The effectiveness of our proposed approach is evaluated and validated through a series of comprehensive experiments using the MISP2021 and MISP2022 datasets. Our code is available at https://github.com/dalision/ModalBiasAVSR

Figures (9)

• Figure 1: CER (in %) degradation curves of AVSR trained with different dropout rates on video frames. Compared with the baseline AVSR without dropout (in red), other AVSR systems perform better with missing input but worse with complete data input. As the training dropout rate increases, the CER curve of AVSR gradually converges to that of ASR (dotted line).
• Figure 2: Two groups of similarity analysis between ASR and AVSR transcriptions. In both groups, an increase in the similarity of recognition transcriptions is observed as the training dropout rate increases. The similarity is measured by relative CER (in %), where the ASR transcription replaces the ground truth.
• Figure 3: Similarity matrices of intermediate representations between ASR and different AVSR settings. As training dropout rates increase, the diagonal lines become brighter, indicating closer proximity between the multimodal and the unimodal distributions of the latent decisive subspace in AVSR.
• Figure 4: An illustration of the Modality Bias Hypothesis (MBH). In the left subplot, the task-relevant component (shaded part) of the latent representations consists of $Z^{sa}$, $Z^{sv}$ and $Z^g$, representing audio-specific, visual-specific decision features and modality-general decisive features respectively. The corresponding proportions are denoted by $\alpha$, $\beta$, and $\gamma$. The right subplot shows a dynamic process of decisive bias with an increasing training dropout rate. Dropout leads to a consistent modality bias on audio, regardless of the extent of the missing.
• Figure 5: Overall framework of the proposed AVSR system. We address challenging real-world scenarios involving missing video frames and noisy speech with an overlap rate exceeding 40% during both the training and testing stages. In MDA-KD, latent knowledge is sampled from the latent distribution of the teacher model with complete data input. This latent konwledege serves as an anchor point to prevent dropout-induced modality bias during the robustness training of the student network. For entirely missing video input, the MS-Adapter is activated to enable a dynamic decision switch.