Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Speaker Encoding Network for Multi-Talker Speech Recognition

Jiawen Kang, Lingwei Meng, Mingyu Cui, Haohan Guo, Xixin Wu, Xunying Liu, Helen Meng

TL;DR

The paper addresses multi-talker ASR by examining limitations of branch-based SIMO and attention-based SISO systems and introducing Cross-Speaker Encoding (CSE). CSE adds a cross-encoder and a joint-HEAT module to enable inter-branch conditioning while preserving a unified output stream, and it is further integrated with Serialized Output Training (CSE-SOT) to blend SIMO and SISO strengths. Experiments on LibriSpeechMix show that CSE improves over SIMO baselines and that CSE-SOT yields substantial gains for overlapping speech, with 10% overall and up to 16% improvements on high-overlap cases; results also demonstrate ablations validating the components and visualization of attention patterns supporting the method. The work demonstrates the viability of hybrid SIMO-SISO approaches for multi-talker ASR and suggests that cross-branch interaction and joint training can better handle overlap while maintaining generalization to more speakers. The approach has practical implications for robust multi-speaker transcription in real-world conversational AI systems.

Abstract

End-to-end multi-talker speech recognition has garnered great interest as an effective approach to directly transcribe overlapped speech from multiple speakers. Current methods typically adopt either 1) single-input multiple-output (SIMO) models with a branched encoder, or 2) single-input single-output (SISO) models based on attention-based encoder-decoder architecture with serialized output training (SOT). In this work, we propose a Cross-Speaker Encoding (CSE) network to address the limitations of SIMO models by aggregating cross-speaker representations. Furthermore, the CSE model is integrated with SOT to leverage both the advantages of SIMO and SISO while mitigating their drawbacks. To the best of our knowledge, this work represents an early effort to integrate SIMO and SISO for multi-talker speech recognition. Experiments on the two-speaker LibrispeechMix dataset show that the CES model reduces word error rate (WER) by 8% over the SIMO baseline. The CSE-SOT model reduces WER by 10% overall and by 16% on high-overlap speech compared to the SOT model. Code is available at https://github.com/kjw11/CSEnet-ASR.

Cross-Speaker Encoding Network for Multi-Talker Speech Recognition

TL;DR

The paper addresses multi-talker ASR by examining limitations of branch-based SIMO and attention-based SISO systems and introducing Cross-Speaker Encoding (CSE). CSE adds a cross-encoder and a joint-HEAT module to enable inter-branch conditioning while preserving a unified output stream, and it is further integrated with Serialized Output Training (CSE-SOT) to blend SIMO and SISO strengths. Experiments on LibriSpeechMix show that CSE improves over SIMO baselines and that CSE-SOT yields substantial gains for overlapping speech, with 10% overall and up to 16% improvements on high-overlap cases; results also demonstrate ablations validating the components and visualization of attention patterns supporting the method. The work demonstrates the viability of hybrid SIMO-SISO approaches for multi-talker ASR and suggests that cross-branch interaction and joint training can better handle overlap while maintaining generalization to more speakers. The approach has practical implications for robust multi-speaker transcription in real-world conversational AI systems.

Abstract

End-to-end multi-talker speech recognition has garnered great interest as an effective approach to directly transcribe overlapped speech from multiple speakers. Current methods typically adopt either 1) single-input multiple-output (SIMO) models with a branched encoder, or 2) single-input single-output (SISO) models based on attention-based encoder-decoder architecture with serialized output training (SOT). In this work, we propose a Cross-Speaker Encoding (CSE) network to address the limitations of SIMO models by aggregating cross-speaker representations. Furthermore, the CSE model is integrated with SOT to leverage both the advantages of SIMO and SISO while mitigating their drawbacks. To the best of our knowledge, this work represents an early effort to integrate SIMO and SISO for multi-talker speech recognition. Experiments on the two-speaker LibrispeechMix dataset show that the CES model reduces word error rate (WER) by 8% over the SIMO baseline. The CSE-SOT model reduces WER by 10% overall and by 16% on high-overlap speech compared to the SOT model. Code is available at https://github.com/kjw11/CSEnet-ASR.
Paper Structure (15 sections, 2 equations, 2 figures, 2 tables)

This paper contains 15 sections, 2 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Cross-Speaker Encoding Network
  3. Branch-based SIMO model
  4. Cross Speaker Encoding network
  5. Integrated CSE-SOT model
  6. Experimental setup
  7. Dataset
  8. Model settings
  9. Training settings & Metrics
  10. Results and Discussions
  11. Results of CSE
  12. Results of CSE-SOT
  13. Visualization
  14. Conclusions
  15. Acknowledgements

Figures (2)

  • Figure 1: The architecture of (a) standard branch-based SIMO model, (b) proposed Cross Speaker Encoding (CSE) network, (c) Integrated model with CSE and serialized output training (SOT), and (4) cross-encoder. SpkrDiff. refers to speaker differentiator, Rec. refers to recognition, and PPE refers to partition-wise positional embedding. $\oplus$ stands for concatenation and $\hat{X}$ stands for mixture encoding.
  • Figure 2: Attention matrices of the last conformer block in the cross-encoder. (a)-(d) refers to 4 attention heads. $\hat{X}$, $S_1$ and $S_2$ denote the input partitions. $\bar{X}$, $\hat{S}_1$ and $\hat{S}_2$ denote the corresponding output partitions.