Listening to Multi-talker Conversations: Modular and End-to-end Perspectives

TL;DR

This work tackles speaker-attributed multi-talker transcription by presenting both a modular pipeline and an end-to-end SURT framework. The modular approach combines overlap-aware diarization, target-speaker extraction via GPU-accelerated guided source separation, and ASR, while the end-to-end SURT model unmixes and transcribes streaming speech with an auxiliary speaker branch for label synchronization. Key contributions include overlap-aware spectral clustering, DOVER-Lap ensembling for diarization, a fast GPU GSS implementation, and SURT variants that support speaker attribution with mechanisms like blank-sharing and speaker prefixing. Across LibriCSS, AMI, ICSI, and AliMeeting, these methods yield meaningful improvements in DER and cpWER, demonstrating practical gains for real meeting transcription and streaming scenarios. The work also provides open-source software that has already influenced community baselines and challenges, underscoring its impact on multi-talker speech processing.

Abstract

Since the first speech recognition systems were built more than 30 years ago, improvement in voice technology has enabled applications such as smart assistants and automated customer support. However, conversation intelligence of the future requires recognizing free-flowing multi-party conversations, which is a crucial and challenging component that still remains unsolved. In this dissertation, we focus on this problem of speaker-attributed multi-talker speech recognition, and propose two perspectives which result from its probabilistic formulation. In the modular perspective, we build a pipeline of sub-tasks involving speaker diarization, target speaker extraction, and speech recognition. Our first contribution is a method to perform overlap-aware diarization by reformulating spectral clustering as a constrained optimization problem. We also describe an algorithm to ensemble diarization outputs, either to combine overlap-aware systems or to perform multi-channel diarization by late fusion. Once speaker segments are identified, we robustly extract single-speaker utterances from the mixture using a GPU-accelerated implementation of guided source separation, which allows us to use an off-the-shelf ASR system to obtain speaker-attributed transcripts. Since the modular approach suffers from error propagation, we propose an alternate "end-to-end" perspective on the problem. For this, we describe the Streaming Unmixing and Recognition Transducer (SURT). We show how to train SURT models efficiently by carefully designing the network architecture, objective functions, and mixture simulation techniques. Finally, we add an auxiliary speaker branch to enable joint prediction of speaker labels synchronized with the speech tokens. We demonstrate that training on synthetic mixtures and adapting with real data helps these models transfer well for streaming transcription of real meeting sessions.

Figures (35)

• Figure 1: Toy examples to demonstrate differences of multi-talker WER definitions, based on Figure 1 in vonNeumann2022OnWE. Each solid box is a word, and grey hatched box is an utterance. Error counts for ORC-WER and cpWER are shown in the tables.
• Figure 2: An illustration of the speaker diarization task. The system estimates $N=3$ speakers in the recording.
• Figure 3: Architecture of the neural network used for frame-level classification for overlap detection.
• Figure 4: T-SNE plots of x-vector embeddings for (a) non-overlapping, and (b) overlapping segments for the recording EN2002a in the AMI eval set (containing 4 speakers). Colors denote the speaker assigned to the segment. For (b), each color represents a distinct pair of speaker labels, resulting in 6 differently colored clusters.
• Figure 5: An illustration of overlapping output produced by DOVER-Lap for overlapping hypotheses.

Theorems & Definitions (36)

• Definition 3.3.1: Turán graph
• Lemma 3.3.1
• Lemma 3.3.2
• proof
• Theorem 3.3.3
• proof
• Definition 3.3.2
• Definition 3.3.3
• Definition 3.3.4
• Definition 3.3.5