Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Serialized Output Training by Learned Dominance

Ying Shi, Lantian Li, Shi Yin, Dong Wang, Jiqing Han

TL;DR

The paper tackles label-permutation in multi-talker ASR and introduces DOM-SOT, a dominance-based serialization that learns to order speaker transcripts by incorporating a CTC-based serialization module into an AED architecture. It analyzes FIFO-SOT, PIT-SOT, and the proposed DOM-SOT, deriving an objective Loss^{dom} = \alpha \cdot \min_{i} {CTC}(h,L_i) + (1-\alpha) \cdot CE(y, L^{\epsilon}) with a small $\alpha$, and demonstrates that DOM-SOT yields significant gains over PIT-SOT and competitive results against FIFO on LibriSpeech/LibriMix. Comprehensive experiments show DOM-SOT advantages are linked to learned biases such as loudness and gender, indicating a robust multi-factor ordering mechanism beyond single-bias baselines. The work also introduces speaker-aware WER to better quantify per-speaker transcription quality, underscoring practical implications for robust multi-talker ASR and guiding future validation on additional datasets.

Abstract

Serialized Output Training (SOT) has showcased state-of-the-art performance in multi-talker speech recognition by sequentially decoding the speech of individual speakers. To address the challenging label-permutation issue, prior methods have relied on either the Permutation Invariant Training (PIT) or the time-based First-In-First-Out (FIFO) rule. This study presents a model-based serialization strategy that incorporates an auxiliary module into the Attention Encoder-Decoder architecture, autonomously identifying the crucial factors to order the output sequence of the speech components in multi-talker speech. Experiments conducted on the LibriSpeech and LibriMix databases reveal that our approach significantly outperforms the PIT and FIFO baselines in both 2-mix and 3-mix scenarios. Further analysis shows that the serialization module identifies dominant speech components in a mixture by factors including loudness and gender, and orders speech components based on the dominance score.

Serialized Output Training by Learned Dominance

TL;DR

The paper tackles label-permutation in multi-talker ASR and introduces DOM-SOT, a dominance-based serialization that learns to order speaker transcripts by incorporating a CTC-based serialization module into an AED architecture. It analyzes FIFO-SOT, PIT-SOT, and the proposed DOM-SOT, deriving an objective Loss^{dom} = \alpha \cdot \min_{i} {CTC}(h,L_i) + (1-\alpha) \cdot CE(y, L^{\epsilon}) with a small , and demonstrates that DOM-SOT yields significant gains over PIT-SOT and competitive results against FIFO on LibriSpeech/LibriMix. Comprehensive experiments show DOM-SOT advantages are linked to learned biases such as loudness and gender, indicating a robust multi-factor ordering mechanism beyond single-bias baselines. The work also introduces speaker-aware WER to better quantify per-speaker transcription quality, underscoring practical implications for robust multi-talker ASR and guiding future validation on additional datasets.

Abstract

Serialized Output Training (SOT) has showcased state-of-the-art performance in multi-talker speech recognition by sequentially decoding the speech of individual speakers. To address the challenging label-permutation issue, prior methods have relied on either the Permutation Invariant Training (PIT) or the time-based First-In-First-Out (FIFO) rule. This study presents a model-based serialization strategy that incorporates an auxiliary module into the Attention Encoder-Decoder architecture, autonomously identifying the crucial factors to order the output sequence of the speech components in multi-talker speech. Experiments conducted on the LibriSpeech and LibriMix databases reveal that our approach significantly outperforms the PIT and FIFO baselines in both 2-mix and 3-mix scenarios. Further analysis shows that the serialization module identifies dominant speech components in a mixture by factors including loudness and gender, and orders speech components based on the dominance score.
Paper Structure (16 sections, 5 equations, 2 figures, 1 table)

This paper contains 16 sections, 5 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. FIFO-SOT & PIT-SOT
  4. DOM-SOT
  5. Experiment Settings
  6. Training Data
  7. Test Data
  8. Model Configuration
  9. Training Strategy
  10. Speaker-Aware WER
  11. Experimental Result
  12. FIFO-SOT vs PIT-SOT
  13. DOM-SOT vs PIT-SOT and FIFO-SOT
  14. Speaker Aware vs Speaker Blind
  15. Analysis & Discussion
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Diagram of FIFO-SOT, PIT-SOT, and our DOM-SOT
  • Figure 2: Proportion of factors contributing to the dominance of dominant speech under the condition 2-mix with 0s/3s offset.