Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AV-CrossNet: an Audiovisual Complex Spectral Mapping Network for Speech Separation By Leveraging Narrow- and Cross-Band Modeling

Vahid Ahmadi Kalkhorani, Cheng Yu, Anurag Kumar, Ke Tan, Buye Xu, DeLiang Wang

TL;DR

AV-CrossNet addresses audiovisual speech separation and target extraction by extending CrossNet with a visual front-end (DeepAVSR) and an early audiovisual fusion. It uses a multi-stage AV-CrossNet block that integrates narrow-band, cross-band, and global self-attention to perform complex spectral mapping of the mixture, estimating real and imaginary spectrograms $\mathbf{S}_c$ from $\mathbf{Y}$ and visual streams $\mathbf{V}_c$. The approach resolves permutation ambiguity via visual cues and demonstrates state-of-the-art results across clean and noisy datasets, as well as untrained conditions, on tasks including AV speech separation, AV target speaker extraction, and AV speech enhancement. The method is computationally efficient, supports half-precision training, and offers robust performance gains on practical audiovisual speech processing applications.

Abstract

Adding visual cues to audio-based speech separation can improve separation performance. This paper introduces AV-CrossNet, an audiovisual (AV) system for speech enhancement, target speaker extraction, and multi-talker speaker separation. AV-CrossNet is extended from the CrossNet architecture, which is a recently proposed network that performs complex spectral mapping for speech separation by leveraging global attention and positional encoding. To effectively utilize visual cues, the proposed system incorporates pre-extracted visual embeddings and employs a visual encoder comprising temporal convolutional layers. Audio and visual features are fused in an early fusion layer before feeding to AV-CrossNet blocks. We evaluate AV-CrossNet on multiple datasets, including LRS, VoxCeleb, and COG-MHEAR challenge. Evaluation results demonstrate that AV-CrossNet advances the state-of-the-art performance in all audiovisual tasks, even on untrained and mismatched datasets.

AV-CrossNet: an Audiovisual Complex Spectral Mapping Network for Speech Separation By Leveraging Narrow- and Cross-Band Modeling

TL;DR

AV-CrossNet addresses audiovisual speech separation and target extraction by extending CrossNet with a visual front-end (DeepAVSR) and an early audiovisual fusion. It uses a multi-stage AV-CrossNet block that integrates narrow-band, cross-band, and global self-attention to perform complex spectral mapping of the mixture, estimating real and imaginary spectrograms from and visual streams . The approach resolves permutation ambiguity via visual cues and demonstrates state-of-the-art results across clean and noisy datasets, as well as untrained conditions, on tasks including AV speech separation, AV target speaker extraction, and AV speech enhancement. The method is computationally efficient, supports half-precision training, and offers robust performance gains on practical audiovisual speech processing applications.

Abstract

Adding visual cues to audio-based speech separation can improve separation performance. This paper introduces AV-CrossNet, an audiovisual (AV) system for speech enhancement, target speaker extraction, and multi-talker speaker separation. AV-CrossNet is extended from the CrossNet architecture, which is a recently proposed network that performs complex spectral mapping for speech separation by leveraging global attention and positional encoding. To effectively utilize visual cues, the proposed system incorporates pre-extracted visual embeddings and employs a visual encoder comprising temporal convolutional layers. Audio and visual features are fused in an early fusion layer before feeding to AV-CrossNet blocks. We evaluate AV-CrossNet on multiple datasets, including LRS, VoxCeleb, and COG-MHEAR challenge. Evaluation results demonstrate that AV-CrossNet advances the state-of-the-art performance in all audiovisual tasks, even on untrained and mismatched datasets.
Paper Structure (28 sections, 11 equations, 4 figures, 4 tables)

This paper contains 28 sections, 11 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Problem formulation
  3. System description
  4. Audio Encoder
  5. Visual Encoder
  6. Audiovisual Fusion
  7. Positional Encoding
  8. AV-CrossNet Block
  9. Narrow-band Module
  10. Cross-band Module
  11. Global Multi-Head Self-Attention Module
  12. Decoder
  13. Loss Function
  14. Experimental Setup
  15. Data Preparation
  16. ...and 13 more sections

Figures (4)

  • Figure 1: Diagram of the proposed AV-CrossNet system for and . For , a video component is added for each speaker. Faces in this figure are blurred for privacy reasons.
  • Figure 2: AV-CrossNet building blocks. (a) Narrow-band module. (b) Cross-band module. (c) Global multi-head self-attention module.
  • Figure 3: SI-SDR results of AV-CrossNet across different SNRs for unprocessed mixtures on the test datasets TCD-TIMIT, LRS3, and VoxCeleb2. Error bars represent 95% confidence intervals.
  • Figure 4: Target speaker extraction results of different gender pairs. The first and second letters denote the genders of the target and interfering speakers, respectively.