Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-View Based Audio Visual Target Speaker Extraction

Peijun Yang, Zhan Jin, Juan Liu, Ming Li

TL;DR

This work proposes Multi-View Tensor Fusion (MVTF), a novel framework that transforms multi-view learning into single-view performance gains and enhances the robustness of AVTSE methods.

Abstract

Audio-Visual Target Speaker Extraction (AVTSE) aims to separate a target speaker's voice from a mixed audio signal using the corresponding visual cues. While most existing AVTSE methods rely exclusively on frontal-view videos, this limitation restricts their robustness in real-world scenarios where non-frontal views are prevalent. Such visual perspectives often contain complementary articulatory information that could enhance speech extraction. In this work, we propose Multi-View Tensor Fusion (MVTF), a novel framework that transforms multi-view learning into single-view performance gains. During the training stage, we leverage synchronized multi-perspective lip videos to learn cross-view correlations through MVTF, where pairwise outer products explicitly model multiplicative interactions between different views of input lip embeddings. At the inference stage, the system supports both single-view and multi-view inputs. Experimental results show that in the single-view inputs, our framework leverages multi-view knowledge to achieve significant performance gains, while in the multi-view mode, it further improves overall performance and enhances the robustness. Our demo, code and data are available at https://anonymous.4open.science/w/MVTF-Gridnet-209C/

Multi-View Based Audio Visual Target Speaker Extraction

TL;DR

This work proposes Multi-View Tensor Fusion (MVTF), a novel framework that transforms multi-view learning into single-view performance gains and enhances the robustness of AVTSE methods.

Abstract

Audio-Visual Target Speaker Extraction (AVTSE) aims to separate a target speaker's voice from a mixed audio signal using the corresponding visual cues. While most existing AVTSE methods rely exclusively on frontal-view videos, this limitation restricts their robustness in real-world scenarios where non-frontal views are prevalent. Such visual perspectives often contain complementary articulatory information that could enhance speech extraction. In this work, we propose Multi-View Tensor Fusion (MVTF), a novel framework that transforms multi-view learning into single-view performance gains. During the training stage, we leverage synchronized multi-perspective lip videos to learn cross-view correlations through MVTF, where pairwise outer products explicitly model multiplicative interactions between different views of input lip embeddings. At the inference stage, the system supports both single-view and multi-view inputs. Experimental results show that in the single-view inputs, our framework leverages multi-view knowledge to achieve significant performance gains, while in the multi-view mode, it further improves overall performance and enhances the robustness. Our demo, code and data are available at https://anonymous.4open.science/w/MVTF-Gridnet-209C/
Paper Structure (17 sections, 5 equations, 1 figure, 4 tables)

This paper contains 17 sections, 5 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Overview
  4. Audio Processing Backbone
  5. Multi-View Feature Extraction
  6. Multi-View Tensor Fusion Module
  7. Experimental Setup
  8. Dataset
  9. Implementation Details
  10. Baseline and Ablation Studies
  11. Results and Discussion
  12. Multi-View Training Enhancement for Single-View Testing
  13. Performance for Multi-View Conditions during Inference
  14. Comparison of Fusion Strategies and Model Complexity
  15. Comparison with previous works on the MEAD dataset
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Our proposed MVTF method that extracts the target speakers' speech using Multi-View lip embeddings. (a)Three views, one view (copied three times), and two views (one view copied two times) visible situations. (b)overall structure of MVTF-GridNet.