Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MSA-ASR: Efficient Multilingual Speaker Attribution with frozen ASR Models

Thai-Binh Nguyen, Alexander Waibel

TL;DR

This work tackles multilingual speaker-attributed automatic speech recognition (SA-ASR) by freezing a large multilingual ASR like Whisper and training a dedicated Speaker module with weak, monolingual data to produce per-token speaker embeddings. The Embedding Alignment and Discrimination (EAD) loss combines cosine-based alignment and intra-/inter-embedding coherence to pair ASR outputs with target speaker embeddings without explicit speaker labels. Across multilingual, mixed-language, and monolingual benchmarks, MSA-ASR shows competitive cpWER relative to strong baselines, excelling in non-overlapping scenarios and offering robustness across languages without per-language fine-tuning. The approach demonstrates practical, scalable multilingual SA-ASR and is released for public use to foster further research.

Abstract

Speaker-attributed automatic speech recognition (SA-ASR) aims to transcribe speech while assigning transcripts to the corresponding speakers accurately. Existing methods often rely on complex modular systems or require extensive fine-tuning of joint modules, limiting their adaptability and general efficiency. This paper introduces a novel approach, leveraging a frozen multilingual ASR model to incorporate speaker attribution into the transcriptions, using only standard monolingual ASR datasets. Our method involves training a speaker module to predict speaker embeddings based on weak labels without requiring additional ASR model modifications. Despite being trained exclusively with non-overlapping monolingual data, our approach effectively extracts speaker attributes across diverse multilingual datasets, including those with overlapping speech. Experimental results demonstrate competitive performance compared to strong baselines, highlighting the model's robustness and potential for practical applications.

MSA-ASR: Efficient Multilingual Speaker Attribution with frozen ASR Models

TL;DR

This work tackles multilingual speaker-attributed automatic speech recognition (SA-ASR) by freezing a large multilingual ASR like Whisper and training a dedicated Speaker module with weak, monolingual data to produce per-token speaker embeddings. The Embedding Alignment and Discrimination (EAD) loss combines cosine-based alignment and intra-/inter-embedding coherence to pair ASR outputs with target speaker embeddings without explicit speaker labels. Across multilingual, mixed-language, and monolingual benchmarks, MSA-ASR shows competitive cpWER relative to strong baselines, excelling in non-overlapping scenarios and offering robustness across languages without per-language fine-tuning. The approach demonstrates practical, scalable multilingual SA-ASR and is released for public use to foster further research.

Abstract

Speaker-attributed automatic speech recognition (SA-ASR) aims to transcribe speech while assigning transcripts to the corresponding speakers accurately. Existing methods often rely on complex modular systems or require extensive fine-tuning of joint modules, limiting their adaptability and general efficiency. This paper introduces a novel approach, leveraging a frozen multilingual ASR model to incorporate speaker attribution into the transcriptions, using only standard monolingual ASR datasets. Our method involves training a speaker module to predict speaker embeddings based on weak labels without requiring additional ASR model modifications. Despite being trained exclusively with non-overlapping monolingual data, our approach effectively extracts speaker attributes across diverse multilingual datasets, including those with overlapping speech. Experimental results demonstrate competitive performance compared to strong baselines, highlighting the model's robustness and potential for practical applications.

Paper Structure

This paper contains 10 sections, 1 equation, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. Modeling
  4. Data Processing
  5. Experiments
  6. Datasets
  7. Modeling and Metric
  8. Results
  9. Conclusion
  10. Acknowledgment

Figures (2)

  • Figure 1: Overview of MSA-ASR model. ASR decoder processes tokens sequentially during inference, while the Speaker decoder can generate speaker embeddings in parallel.
  • Figure 2: Speaker decoder architecture. Similar as standard transformer decoder, but cross-attention uses key and value from a different encoders.