Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition

Sungnyun Kim, Kangwook Jang, Sangmin Bae, Hoirin Kim, Se-Young Yun

TL;DR

This work strengthens the video features by learning three temporal dynamics in video data: context order, playback direction, and the speed of video frames, and introduces cross-modal attention modules to enrich video features with audio information so that speech variability can be taken into account when training on the video temporal dynamics.

Abstract

Audio-visual speech recognition (AVSR) aims to transcribe human speech using both audio and video modalities. In practical environments with noise-corrupted audio, the role of video information becomes crucial. However, prior works have primarily focused on enhancing audio features in AVSR, overlooking the importance of video features. In this study, we strengthen the video features by learning three temporal dynamics in video data: context order, playback direction, and the speed of video frames. Cross-modal attention modules are introduced to enrich video features with audio information so that speech variability can be taken into account when training on the video temporal dynamics. Based on our approach, we achieve the state-of-the-art performance on the LRS2 and LRS3 AVSR benchmarks for the noise-dominant settings. Our approach excels in scenarios especially for babble and speech noise, indicating the ability to distinguish the speech signal that should be recognized from lip movements in the video modality. We support the validity of our methodology by offering the ablation experiments for the temporal dynamics losses and the cross-modal attention architecture design.

Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition

TL;DR

This work strengthens the video features by learning three temporal dynamics in video data: context order, playback direction, and the speed of video frames, and introduces cross-modal attention modules to enrich video features with audio information so that speech variability can be taken into account when training on the video temporal dynamics.

Abstract

Audio-visual speech recognition (AVSR) aims to transcribe human speech using both audio and video modalities. In practical environments with noise-corrupted audio, the role of video information becomes crucial. However, prior works have primarily focused on enhancing audio features in AVSR, overlooking the importance of video features. In this study, we strengthen the video features by learning three temporal dynamics in video data: context order, playback direction, and the speed of video frames. Cross-modal attention modules are introduced to enrich video features with audio information so that speech variability can be taken into account when training on the video temporal dynamics. Based on our approach, we achieve the state-of-the-art performance on the LRS2 and LRS3 AVSR benchmarks for the noise-dominant settings. Our approach excels in scenarios especially for babble and speech noise, indicating the ability to distinguish the speech signal that should be recognized from lip movements in the video modality. We support the validity of our methodology by offering the ablation experiments for the temporal dynamics losses and the cross-modal attention architecture design.
Paper Structure (18 sections, 8 equations, 2 figures, 4 tables)

This paper contains 18 sections, 8 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Audio-Visual Speech Recognition
  4. Temporal Dynamics Learning
  5. Methodology
  6. Overview
  7. A2V Video Temporal Dynamics Guidance
  8. V2A Audio Refinement
  9. Experiments and Results
  10. Implementation Details
  11. Robust AVSR Benchmark Results
  12. Ablation Study
  13. Conclusion
  14. Acknowledgements
  15. Implementation Details
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Our proposed temporal dynamics guidance ($\mathcal{L}_\text{temp}$) involves predicting (a) the context order considering both video (V) and audio (A) modalities ($\mathcal{L}_\text{order}$; Eq. \ref{['eq:order-loss']}), (b) playback direction ($\mathcal{L}_\text{direction}$; Eq. \ref{['eq:direction-loss']}), and (c) whether certain frames are skipped or not ($\mathcal{L}_\text{speed}$; Eq. \ref{['eq:speed-loss']}). Each video temporal predictor is consisted of 1D convolution and fully-connected (FC) layers.
  • Figure 2: Our cross-modal attention structure is inserted between the feature extractors and the AVSR encoder. This structure leverages clean video to refine audio, and then learns video temporal dynamics given the refined audio features. Note that the gradient is not backpropagated between the two modalities so that training $\mathcal{L}_\text{temp}$ and $\mathcal{L}_\text{ref}$ does not interfere each other.