Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Joint Speaker Features Learning for Audio-visual Multichannel Speech Separation and Recognition

Guinan Li, Jiajun Deng, Youjun Chen, Mengzhe Geng, Shujie Hu, Zhe Li, Zengrui Jin, Tianzi Wang, Xurong Xie, Helen Meng, Xunying Liu

TL;DR

The paper addresses cocktail-party speech with audio-visual multichannel systems and zero-shot adaptation. It introduces joint speaker feature learning by tightly integrating xVector or ECAPA-TDNN encoders with the end-to-end separation-recognition pipeline through tailored fusion blocks, enabling enrollment-free adaptation without pre-recorded data. Empirical results on simulated LRS3-TED mixtures show consistent gains in separation and recognition, with improvements correlating to increased inter-speaker discrimination, and a best system surpasses a strong WavLM baseline when combined with SSL and video features. The approach offers practical benefits for on-device, privacy-conscious, zero-shot personalization in complex acoustic scenes.

Abstract

This paper proposes joint speaker feature learning methods for zero-shot adaptation of audio-visual multichannel speech separation and recognition systems. xVector and ECAPA-TDNN speaker encoders are connected using purpose-built fusion blocks and tightly integrated with the complete system training. Experiments conducted on LRS3-TED data simulated multichannel overlapped speech suggest that joint speaker feature learning consistently improves speech separation and recognition performance over the baselines without joint speaker feature estimation. Further analyses reveal performance improvements are strongly correlated with increased inter-speaker discrimination measured using cosine similarity. The best-performing joint speaker feature learning adapted system outperformed the baseline fine-tuned WavLM model by statistically significant WER reductions of 21.6% and 25.3% absolute (67.5% and 83.5% relative) on Dev and Test sets after incorporating WavLM features and video modality.

Joint Speaker Features Learning for Audio-visual Multichannel Speech Separation and Recognition

TL;DR

The paper addresses cocktail-party speech with audio-visual multichannel systems and zero-shot adaptation. It introduces joint speaker feature learning by tightly integrating xVector or ECAPA-TDNN encoders with the end-to-end separation-recognition pipeline through tailored fusion blocks, enabling enrollment-free adaptation without pre-recorded data. Empirical results on simulated LRS3-TED mixtures show consistent gains in separation and recognition, with improvements correlating to increased inter-speaker discrimination, and a best system surpasses a strong WavLM baseline when combined with SSL and video features. The approach offers practical benefits for on-device, privacy-conscious, zero-shot personalization in complex acoustic scenes.

Abstract

This paper proposes joint speaker feature learning methods for zero-shot adaptation of audio-visual multichannel speech separation and recognition systems. xVector and ECAPA-TDNN speaker encoders are connected using purpose-built fusion blocks and tightly integrated with the complete system training. Experiments conducted on LRS3-TED data simulated multichannel overlapped speech suggest that joint speaker feature learning consistently improves speech separation and recognition performance over the baselines without joint speaker feature estimation. Further analyses reveal performance improvements are strongly correlated with increased inter-speaker discrimination measured using cosine similarity. The best-performing joint speaker feature learning adapted system outperformed the baseline fine-tuned WavLM model by statistically significant WER reductions of 21.6% and 25.3% absolute (67.5% and 83.5% relative) on Dev and Test sets after incorporating WavLM features and video modality.
Paper Structure (14 sections, 3 equations, 2 figures, 2 tables)

This paper contains 14 sections, 3 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Audio-visual Speech Separation
  3. Mask-based MVDR
  4. Audio and Visual Modality Fusion
  5. Speaker Adaptation of Audio-visual Speech Separation Front-end
  6. Speaker Feature Fusion
  7. Speaker Feature Learning
  8. Adaptation Supervision
  9. Audio-visual Conformer ASR Back-end
  10. Experiments
  11. Experimental Setup
  12. Experimental Results
  13. Conclusions
  14. Acknowledgements

Figures (2)

  • Figure 1: Example of joint speaker features learning for an audio-visual multichannel speech separation and recognition system including the following components: (1) Speaker-adaptive speech separation front-end implemented using TCNs and mask-based MVDR beamforming; (2) Conformer ASR Back-end; and (3) Visual Feature extraction Module. The Speaker Encoder (iVector, xVector or ECAPA-TDNN) module is connected with the backbone system using purpose-built fusion blocks based on either (a) Input Bias or (b) Activation Scaling, and tightly integrated with complete system training. Speaker adaptation is performed in either (c) enrollment-based modevzmolikova2019speakerbeamochiai2019multimodalWang2019voicefilterju2022teaju2023teaxu2020spexge2020spex+eskimez2022personalizedtaherian2022one requiring pre-recorded speaker-level parallel clean-noisy speech; or (d) zero-short, enrollment-free mode that does not require pre-recorded speaker-level clean speech data. "Cat” and "$\odot$” denote the concatenation and element-wise product operation, respectively.
  • Figure 2: Correlation between cosine similarity and SISNR or overall WER on "Dev" set for systems in Table \ref{['tab:table2']}. Comparable systems with/without joint and non-joint speaker features learning marked as "$\triangle$" and "$\bigcirc$" respectively at two ends of each colored line. "EF", "E", "IB", and "AS" denote "enrollment free", "enrollment", "input bias" and "activation scaling".