Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Two-stage Audio-Visual Target Speaker Extraction System for Real-Time Processing On Edge Device

Zixuan Li, Xueliang Zhang, Lei Miao, Zhipeng Yan, Ying Sun, Chong Zhu

TL;DR

This work tackles real-time audio-visual target speaker extraction (AVTSE) on edge devices by introducing a two-stage, decoupled framework (2S-AVTSE) that first performs visual voice activity detection (VVAD) and then conducts target speech extraction (TSE) using the VVAD cue. The VVAD front-end is lightweight (0.18 GMACs, ~1.36M parameters) and is trained with a two-stage data strategy, including a synthetic, balanced lip-portrait dataset and VAD augmentation to mimic upstream errors. The TSE backbone (Cross-Narrow Band) uses a GTCRN-based encoder, Cross-Band and Narrow-Band modules with Chunk Attention to achieve linear-time complexity, outputting a 4-channel complex ratio mask for reconstruction via iSTFT; the system achieves real-time inference on standard laptops (1.46 ms on M1 Pro, 2.9 ms on i5) and demonstrates strong generalization on realistic FaceStar-Mix data (SI-SNR = 7.09 dB) while outperforming audio-only anchors. The work highlights the practical viability of AVTSE for online conferencing and edge devices, reducing dependency on synchronized audio-visual data and avoiding user enrollment of anchors, with future work aimed at improving VVAD accuracy and end-to-end integration.

Abstract

Audio-Visual Target Speaker Extraction (AVTSE) aims to isolate a target speaker's voice in a multi-speaker environment with visual cues as auxiliary. Most of the existing AVTSE methods encode visual and audio features simultaneously, resulting in extremely high computational complexity and making it impractical for real-time processing on edge devices. To tackle this issue, we proposed a two-stage ultra-compact AVTSE system. Specifically, in the first stage, a compact network is employed for voice activity detection (VAD) using visual information. In the second stage, the VAD results are combined with audio inputs to isolate the target speaker's voice. Experiments show that the proposed system effectively suppresses background noise and interfering voices while spending little computational resources.

Two-stage Audio-Visual Target Speaker Extraction System for Real-Time Processing On Edge Device

TL;DR

This work tackles real-time audio-visual target speaker extraction (AVTSE) on edge devices by introducing a two-stage, decoupled framework (2S-AVTSE) that first performs visual voice activity detection (VVAD) and then conducts target speech extraction (TSE) using the VVAD cue. The VVAD front-end is lightweight (0.18 GMACs, ~1.36M parameters) and is trained with a two-stage data strategy, including a synthetic, balanced lip-portrait dataset and VAD augmentation to mimic upstream errors. The TSE backbone (Cross-Narrow Band) uses a GTCRN-based encoder, Cross-Band and Narrow-Band modules with Chunk Attention to achieve linear-time complexity, outputting a 4-channel complex ratio mask for reconstruction via iSTFT; the system achieves real-time inference on standard laptops (1.46 ms on M1 Pro, 2.9 ms on i5) and demonstrates strong generalization on realistic FaceStar-Mix data (SI-SNR = 7.09 dB) while outperforming audio-only anchors. The work highlights the practical viability of AVTSE for online conferencing and edge devices, reducing dependency on synchronized audio-visual data and avoiding user enrollment of anchors, with future work aimed at improving VVAD accuracy and end-to-end integration.

Abstract

Audio-Visual Target Speaker Extraction (AVTSE) aims to isolate a target speaker's voice in a multi-speaker environment with visual cues as auxiliary. Most of the existing AVTSE methods encode visual and audio features simultaneously, resulting in extremely high computational complexity and making it impractical for real-time processing on edge devices. To tackle this issue, we proposed a two-stage ultra-compact AVTSE system. Specifically, in the first stage, a compact network is employed for voice activity detection (VAD) using visual information. In the second stage, the VAD results are combined with audio inputs to isolate the target speaker's voice. Experiments show that the proposed system effectively suppresses background noise and interfering voices while spending little computational resources.

Paper Structure

This paper contains 16 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. 2S-AVTSE
  3. First Stage: Visual Voice Activity Detection
  4. Second Stage: Target Speaker Extraction
  5. Encoder
  6. Cross-Narrow Band Backbone
  7. Decoder
  8. Experiments
  9. VVAD Module Data Preparation
  10. TSE Module Data Preparation
  11. Evaluation
  12. Performance on LRS2-2mix
  13. Performance on Realistic and Real-World Data
  14. Ablation Studies
  15. Ease of deployment
  16. ...and 1 more sections

Figures (2)

  • Figure 1: overview of the 2S-AVTSE architecture along with detailed structures of its individual components. (a) Overview of 2S-AVTSE, VVAD module is ths first stage, TSE module is the second stage. (b) Overview of VVAD Module. (c) Overview of the TSE Module, where $\odot$ represents element-wise multiplication. (d) Overview of Cross-Band Module. (e) Overview of Narrow-Band Module.
  • Figure 2: Real recording and the outputs of different methods.