Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Plug-and-Steer: Decoupling Separation and Selection in Audio-Visual Target Speaker Extraction

Doyeop Kwak, Suyeon Lee, Joon Son Chung

Abstract

The goal of this paper is to provide a new perspective on audio-visual target speaker extraction (AV-TSE) by decoupling the separation and target selection. Conventional AV-TSE systems typically integrate audio and visual features deeply to re-learn the entire separation process, which can act as a fidelity ceiling due to the noisy nature of in-the-wild audio-visual datasets. To address this, we propose Plug-and-Steer, which assigns high-fidelity separation to a frozen audio-only backbone and limits the role of visual modality strictly to target selection. We introduce the Latent Steering Matrix (LSM), a minimalist linear transformation that re-routes latent features within the backbone to anchor the target speaker to a designated channel. Experiments across four representative architectures show that our method effectively preserves the acoustic priors of diverse backbones, achieving perceptual quality comparable to the original backbones. Audio samples are available at: https://plugandsteer.github.io

Plug-and-Steer: Decoupling Separation and Selection in Audio-Visual Target Speaker Extraction

Abstract

The goal of this paper is to provide a new perspective on audio-visual target speaker extraction (AV-TSE) by decoupling the separation and target selection. Conventional AV-TSE systems typically integrate audio and visual features deeply to re-learn the entire separation process, which can act as a fidelity ceiling due to the noisy nature of in-the-wild audio-visual datasets. To address this, we propose Plug-and-Steer, which assigns high-fidelity separation to a frozen audio-only backbone and limits the role of visual modality strictly to target selection. We introduce the Latent Steering Matrix (LSM), a minimalist linear transformation that re-routes latent features within the backbone to anchor the target speaker to a designated channel. Experiments across four representative architectures show that our method effectively preserves the acoustic priors of diverse backbones, achieving perceptual quality comparable to the original backbones. Audio samples are available at: https://plugandsteer.github.io
Paper Structure (18 sections, 3 equations, 3 figures, 4 tables)

This paper contains 18 sections, 3 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Latent Steering Matrix (LSM): A steering wheel
  4. Training LSM
  5. Visual steering module: Learning to steer
  6. Experimental Setup
  7. Datasets
  8. Training and evaluation details
  9. Alternative adaptation strategies for AV-TSE
  10. Results and Anaylsis
  11. Layer-wise performance preservation
  12. AV-TSE performance
  13. Analysis of acoustic prior preservation
  14. Scaling with powerful AO backbones
  15. In-domain adaptation
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Training process of the latent steering matrix $W$ for the $i$-th separator block. $\hat{s}_1$ and $\hat{s}_2$ are the outputs from the first and second channels of the pre-trained audio-only backbone.
  • Figure 2: Training process of the AV-TSE module by learning the gate value $g$. The visual steering module predicts the value $g$ to control the degree of steering.
  • Figure 3: Layer-wise performance preservation rate (%) across different AOSS backbones.