Adapting Diarization-Conditioned Whisper for End-to-End Multi-Talker Speech Recognition
Martin Kocour, Martin Karafiat, Alexander Polok, Dominik Klement, Lukáš Burget, Jan Černocký
TL;DR
This work tackles multi-talker ASR with explicit speaker attribution by marrying target-speaker conditioning with serialized output training in a Whisper-based framework. It introduces SA-DiCoW, which uses a DiCoW encoder to generate per-speaker embeddings $\hat{\mathbf{H}}_u$ that are affine-transformed and concatenated into a joint encoder representation $\bar{\mathbf{H}}$, enabling a single decoder to produce speaker-tagged transcripts with timestamps via serialized tokens. Empirical results show SA-DiCoW outperforms existing SOT approaches on synthetic LibriMix mixtures and closely rivals or exceeds several prior speaker-attributed systems on real conversational datasets like AMI and NOTSOFAR, albeit with some scenarios where separate decoding remains advantageous. The work also provides insights into aggregation choices, speaker-label assignment, and cross-attention behavior, highlighting the potential and limitations of joint decoding for highly overlapped speech in practical settings.
Abstract
We propose a speaker-attributed (SA) Whisper-based model for multi-talker speech recognition that combines target-speaker modeling with serialized output training (SOT). Our approach leverages a Diarization-Conditioned Whisper (DiCoW) encoder to extract target-speaker embeddings, which are concatenated into a single representation and passed to a shared decoder. This enables the model to transcribe overlapping speech as a serialized output stream with speaker tags and timestamps. In contrast to target-speaker ASR systems such as DiCoW, which decode each speaker separately, our approach performs joint decoding, allowing the decoder to condition on the context of all speakers simultaneously. Experiments show that the model outperforms existing SOT-based approaches and surpasses DiCoW on multi-talker mixtures (e.g., LibriMix).