Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Wake Word Spotting With Frame-Level Cross-Modal Attention Based Audio-Visual Conformer

Haoxu Wang, Ming Cheng, Qiang Fu, Ming Li

TL;DR

This work targets robust wake word spotting in far-field, noisy environments by leveraging lip movements as a cross-modal cue. It introduces Frame-Level Cross-Modal Attention (FLCMA) within an end-to-end Audio-Visual Conformer, along with a pretraining strategy that transfers uni-modal weights to the multimodal model. The approach achieves state-of-the-art performance on the far-field MISP AVWWS dataset, with a WWS score of 4.57%, demonstrating improved robustness through frame-level inter-modal modeling and pretraining. The results highlight the practical potential for more reliable wake word detection in real-world conditions where audio alone is insufficient.

Abstract

In recent years, neural network-based Wake Word Spotting achieves good performance on clean audio samples but struggles in noisy environments. Audio-Visual Wake Word Spotting (AVWWS) receives lots of attention because visual lip movement information is not affected by complex acoustic scenes. Previous works usually use simple addition or concatenation for multi-modal fusion. The inter-modal correlation remains relatively under-explored. In this paper, we propose a novel module called Frame-Level Cross-Modal Attention (FLCMA) to improve the performance of AVWWS systems. This module can help model multi-modal information at the frame-level through synchronous lip movements and speech signals. We train the end-to-end FLCMA based Audio-Visual Conformer and further improve the performance by fine-tuning pre-trained uni-modal models for the AVWWS task. The proposed system achieves a new state-of-the-art result (4.57% WWS score) on the far-field MISP dataset.

Robust Wake Word Spotting With Frame-Level Cross-Modal Attention Based Audio-Visual Conformer

TL;DR

This work targets robust wake word spotting in far-field, noisy environments by leveraging lip movements as a cross-modal cue. It introduces Frame-Level Cross-Modal Attention (FLCMA) within an end-to-end Audio-Visual Conformer, along with a pretraining strategy that transfers uni-modal weights to the multimodal model. The approach achieves state-of-the-art performance on the far-field MISP AVWWS dataset, with a WWS score of 4.57%, demonstrating improved robustness through frame-level inter-modal modeling and pretraining. The results highlight the practical potential for more reliable wake word detection in real-world conditions where audio alone is insufficient.

Abstract

In recent years, neural network-based Wake Word Spotting achieves good performance on clean audio samples but struggles in noisy environments. Audio-Visual Wake Word Spotting (AVWWS) receives lots of attention because visual lip movement information is not affected by complex acoustic scenes. Previous works usually use simple addition or concatenation for multi-modal fusion. The inter-modal correlation remains relatively under-explored. In this paper, we propose a novel module called Frame-Level Cross-Modal Attention (FLCMA) to improve the performance of AVWWS systems. This module can help model multi-modal information at the frame-level through synchronous lip movements and speech signals. We train the end-to-end FLCMA based Audio-Visual Conformer and further improve the performance by fine-tuning pre-trained uni-modal models for the AVWWS task. The proposed system achieves a new state-of-the-art result (4.57% WWS score) on the far-field MISP dataset.
Paper Structure (19 sections, 3 equations, 5 figures, 3 tables)

This paper contains 19 sections, 3 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Frame-Level Cross-Modal Attention
  4. Pretrain Strategy
  5. E2E Model Architecture
  6. Front-end
  7. Encoder
  8. Convolution fusion
  9. Attentive Pooling and Classifier
  10. Experimental Setup
  11. Dataset and Evaluation Metrics
  12. Experimental Setup
  13. Results and Discussions
  14. Ablation Study
  15. Results of the Pretrained single-modality model
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Diagram of simple fusion method (Late: top, Early: middle) and our proposed inter-modality fusion method (bottom). $B$, $T$ and $D$ denote the batch size, the number of frames and the dimension of the embeddings. $M$ represents the number of modalities.
  • Figure 2: (a) Framework of our FLCMA based audio-visual conformer wake word spotting system. (b) Diagram of FLCMA module. Each circle means audio or visual feature embedding at each time step.
  • Figure 3: Framework of convolution fusion module.
  • Figure 4: Visualization of the attention weights of the FLCMA based Audio-Viusal Conformer model.
  • Figure 5: Visualization of attention maps in the FLCMA module of the last encoder conformer block. The head means the four head in the FLCMA module.