Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder

TL;DR

The paper tackles limited gains in end-to-end AVSR with low-quality videos by addressing cross-modal learning dynamics through two innovations: a visual pre-training method that aligns lip shapes with Mandarin syllable subword units via frame-level HMM states, and an audio-dominated Cross-Modal Fusion Encoder (CMFE) that performs multiple cross-attention fusions across layers. Training is performed in two stages: unimodal pre-training and multi-modal fine-tuning, optimizing a joint loss $L_{MTL}=\lambda \log P_{ctc}(Y|X)+(1-\lambda) \log P_{att}(Y|X)$ with $\lambda=0.3$, enabling effective integration of audio and visual streams. Empirical results on the MISP2021-AVSR corpus show that fine-grained syllable-aligned visual pre-training (senone states) and the CMFE approach yield state-of-the-art CERs with relatively small training data, outperforming systems with more complex front-ends/back-ends. This work demonstrates improved lip-visual alignment and cross-modal interaction, providing practical gains for AVSR in real-world noisy video settings. $L_{MTL} = \lambda \log P_{ctc}(Y|X) + (1-\lambda) \log P_{att}(Y|X)$ with $\lambda=0.3$ encapsulates the training objective across modalities.$

Abstract

In recent research, slight performance improvement is observed from automatic speech recognition systems to audio-visual speech recognition systems in the end-to-end framework with low-quality videos. Unmatching convergence rates and specialized input representations between audio and visual modalities are considered to cause the problem. In this paper, we propose two novel techniques to improve audio-visual speech recognition (AVSR) under a pre-training and fine-tuning training framework. First, we explore the correlation between lip shapes and syllable-level subword units in Mandarin to establish good frame-level syllable boundaries from lip shapes. This enables accurate alignment of video and audio streams during visual model pre-training and cross-modal fusion. Next, we propose an audio-guided cross-modal fusion encoder (CMFE) neural network to utilize main training parameters for multiple cross-modal attention layers to make full use of modality complementarity. Experiments on the MISP2021-AVSR data set show the effectiveness of the two proposed techniques. Together, using only a relatively small amount of training data, the final system achieves better performances than state-of-the-art systems with more complex front-ends and back-ends.

Figures (3)

• Figure 1: Overall training framework of our AVSR system
• Figure 2: Cross-modal fusion encoder.
• Figure 3: Resnet output embedding projection through t-SNE from AV7 (left) and AV5 (right)