Target Speaker ASR with Whisper

TL;DR

The paper tackles adapting large single-speaker ASR models to multi-speaker conversations without full separation pipelines by conditioning on diarization outputs. It introduces Diarization-Conditioned Whisper with Frame-Level Diarization Dependent Transformations (FDDT) that apply class-specific affine transformations guided by frame-level diarization cues to distinguish target from non-target speech. A fixed STNO (Silence, Target, Non-target, Overlap) masking framework supports target-focused representations, complemented by an input-masking approach and initialization strategies to preserve pre-trained features. Empirically, the method yields substantial tcORC-WER improvements over baselines (up to 12.9 percentage points on NOTSOFAR-1) and strong results on AMI and Libri2Mix, demonstrating practical TS-ASR capabilities with real-world diarization noise and suggesting avenues for scaling and extension to other pre-trained ASR models.

Abstract

We propose a novel approach to enable the use of large, single-speaker ASR models, such as Whisper, for target speaker ASR. The key claim of this method is that it is much easier to model relative differences among speakers by learning to condition on frame-level diarization outputs than to learn the space of all speaker embeddings. We find that adding even a single bias term per diarization output type before the first transformer block can transform single-speaker ASR models into target-speaker ASR models. Our approach also supports speaker-attributed ASR by sequentially generating transcripts for each speaker in a diarization output. This simplified method outperforms baseline speech separation and diarization cascade by 12.9 % absolute ORC-WER on the NOTSOFAR-1 dataset.

Figures (1)

• Figure 1: Proposed Diarization-Conditioned Whisper model: An input audio segment with multiple speakers is conditioned by frame-level diarization outputs $p_{\mathcal{S}}^tp_{\mathcal{T}}^tp_{\mathcal{N}}^tp_{\mathcal{O}}^t^T$ for each STNO class at every frame $t$. Affine transformations, applied to intermediate input representations $\mathbf{z}^l_{1:T}$, generate new embeddings, where $l$ is the layer index. The final frame-level embedding is a convex combination of these embeddings for each frame.