Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SA-SOT: Speaker-Aware Serialized Output Training for Multi-Talker ASR

Zhiyun Fan, Linhao Dong, Jun Zhang, Lu Lu, Zejun Ma

TL;DR

The paper tackles multi-talker ASR by addressing semantic confusion in token-level serialized output training (t-SOT). It introduces SA-SOT, which integrates token-level speaker embeddings into the ASR decoder (speaker embedding fusion) and applies a speaker similarity-guided self-attention (speaker-aware attention) to strengthen same-speaker context while suppressing cross-speaker interactions; it also employs masked t-SOT labels (SAT) to guide token-attribution. The approach uses a CIF-based ASR backbone with an auxiliary speaker branch for embeddings and training losses that jointly optimize ASR and speaker discrimination. Experiments on Librispeech and LibrispeechMix show substantial relative cpWER reductions on multi-talker test sets (up to $22.03\%$) and state-of-the-art performance ($3.41\%$ cpWER) on LibrispeechMix with extensive training, while preserving single-talker performance. These results demonstrate that incorporating speaker-aware representations and masked auxiliary supervision can markedly improve robustness to overlapped speech in end-to-end ASR models.

Abstract

Multi-talker automatic speech recognition plays a crucial role in scenarios involving multi-party interactions, such as meetings and conversations. Due to its inherent complexity, this task has been receiving increasing attention. Notably, the serialized output training (SOT) stands out among various approaches because of its simplistic architecture and exceptional performance. However, the frequent speaker changes in token-level SOT (t-SOT) present challenges for the autoregressive decoder in effectively utilizing context to predict output sequences. To address this issue, we introduce a masked t-SOT label, which serves as the cornerstone of an auxiliary training loss. Additionally, we utilize a speaker similarity matrix to refine the self-attention mechanism of the decoder. This strategic adjustment enhances contextual relationships within the same speaker's tokens while minimizing interactions between different speakers' tokens. We denote our method as speaker-aware SOT (SA-SOT). Experiments on the Librispeech datasets demonstrate that our SA-SOT obtains a relative cpWER reduction ranging from 12.75% to 22.03% on the multi-talker test sets. Furthermore, with more extensive training, our method achieves an impressive cpWER of 3.41%, establishing a new state-of-the-art result on the LibrispeechMix dataset.

SA-SOT: Speaker-Aware Serialized Output Training for Multi-Talker ASR

TL;DR

The paper tackles multi-talker ASR by addressing semantic confusion in token-level serialized output training (t-SOT). It introduces SA-SOT, which integrates token-level speaker embeddings into the ASR decoder (speaker embedding fusion) and applies a speaker similarity-guided self-attention (speaker-aware attention) to strengthen same-speaker context while suppressing cross-speaker interactions; it also employs masked t-SOT labels (SAT) to guide token-attribution. The approach uses a CIF-based ASR backbone with an auxiliary speaker branch for embeddings and training losses that jointly optimize ASR and speaker discrimination. Experiments on Librispeech and LibrispeechMix show substantial relative cpWER reductions on multi-talker test sets (up to ) and state-of-the-art performance ( cpWER) on LibrispeechMix with extensive training, while preserving single-talker performance. These results demonstrate that incorporating speaker-aware representations and masked auxiliary supervision can markedly improve robustness to overlapped speech in end-to-end ASR models.

Abstract

Multi-talker automatic speech recognition plays a crucial role in scenarios involving multi-party interactions, such as meetings and conversations. Due to its inherent complexity, this task has been receiving increasing attention. Notably, the serialized output training (SOT) stands out among various approaches because of its simplistic architecture and exceptional performance. However, the frequent speaker changes in token-level SOT (t-SOT) present challenges for the autoregressive decoder in effectively utilizing context to predict output sequences. To address this issue, we introduce a masked t-SOT label, which serves as the cornerstone of an auxiliary training loss. Additionally, we utilize a speaker similarity matrix to refine the self-attention mechanism of the decoder. This strategic adjustment enhances contextual relationships within the same speaker's tokens while minimizing interactions between different speakers' tokens. We denote our method as speaker-aware SOT (SA-SOT). Experiments on the Librispeech datasets demonstrate that our SA-SOT obtains a relative cpWER reduction ranging from 12.75% to 22.03% on the multi-talker test sets. Furthermore, with more extensive training, our method achieves an impressive cpWER of 3.41%, establishing a new state-of-the-art result on the LibrispeechMix dataset.
Paper Structure (14 sections, 13 equations, 3 figures, 2 tables)

This paper contains 14 sections, 13 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Framework
  4. Speaker embedding fusion
  5. Speaker-aware training
  6. Speaker-aware attention
  7. Training loss
  8. Experiments
  9. Datasets and metric
  10. Experimental Setup
  11. Main results
  12. Encoder downsampling
  13. Example analysis
  14. Conclusion

Figures (3)

  • Figure 1: An example of the decoding process of t-SOT-based ASR model. For simplicity, the encoder is omitted. The full t-SOT label is 'hello how $<$cc$>$ I $<$cc$>$ are $<$cc$>$ am $<$cc$>$ you $<$cc$>$ fine'.
  • Figure 2: The upper of the figure is the overview of our SA-SOT-based method. ASR branch is colored by blue, the speaker branch is colored by yellow. The lower part is an example showing the labels used in the training.
  • Figure 3: An example of model outputs and reference from the LibrispeechMix. The results for 'spk1' and 'spk2' are obtained by separating the raw hypotheses with the special symbol '$<$cc$>$'.