Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SE-DiCoW: Self-Enrolled Diarization-Conditioned Whisper

Alexander Polok, Dominik Klement, Samuele Cornell, Matthew Wiesner, Jan Černocký, Sanjeev Khudanpur, Lukáš Burget

TL;DR

The paper tackles target-speaker ASR in multi-speaker environments and the STNO conditioning ambiguity that arises in fully overlapped speech. It proposes SE-DiCoW, which automatically self-enrolls a target reference segment by maximizing $ abla \,\sum_{t=start}^{end} p^t_{ ext{T}}$ and fuses it into the encoder via cross-attention, while training losses remain on the main mix. The approach is complemented by DiCoW improvements such as a Pre-Positional FDDT layer, revised initialization, and extensive data augmentation and improved data segmentation. Empirical results show a macro tcpWER reduction of $52.4\%$ relative to the original DiCoW on EMMA MT-ASR and state-of-the-art or competitive performance on AMI SDM and Libri3Mix, under both oracle and real diarization. These findings indicate SE-DiCoW's practical viability for cross-domain TS-ASR and guide future work on joint diarization-TS-ASR training.

Abstract

Speaker-attributed automatic speech recognition (ASR) in multi-speaker environments remains a major challenge. While some approaches achieve strong performance when fine-tuned on specific domains, few systems generalize well across out-of-domain datasets. Our prior work, Diarization-Conditioned Whisper (DiCoW), leverages speaker diarization outputs as conditioning information and, with minimal fine-tuning, demonstrated strong multilingual and multi-domain performance. In this paper, we address a key limitation of DiCoW: ambiguity in Silence-Target-Non-target-Overlap (STNO) masks, where two or more fully overlapping speakers may have nearly identical conditioning despite differing transcriptions. We introduce SE-DiCoW (Self-Enrolled Diarization-Conditioned Whisper), which uses diarization output to locate an enrollment segment anywhere in the conversation where the target speaker is most active. This enrollment segment is used as fixed conditioning via cross-attention at each encoder layer. We further refine DiCoW with improved data segmentation, model initialization, and augmentation. Together, these advances yield substantial gains: SE-DiCoW reduces macro-averaged tcpWER by 52.4% relative to the original DiCoW on the EMMA MT-ASR benchmark.

SE-DiCoW: Self-Enrolled Diarization-Conditioned Whisper

TL;DR

The paper tackles target-speaker ASR in multi-speaker environments and the STNO conditioning ambiguity that arises in fully overlapped speech. It proposes SE-DiCoW, which automatically self-enrolls a target reference segment by maximizing and fuses it into the encoder via cross-attention, while training losses remain on the main mix. The approach is complemented by DiCoW improvements such as a Pre-Positional FDDT layer, revised initialization, and extensive data augmentation and improved data segmentation. Empirical results show a macro tcpWER reduction of relative to the original DiCoW on EMMA MT-ASR and state-of-the-art or competitive performance on AMI SDM and Libri3Mix, under both oracle and real diarization. These findings indicate SE-DiCoW's practical viability for cross-domain TS-ASR and guide future work on joint diarization-TS-ASR training.

Abstract

Speaker-attributed automatic speech recognition (ASR) in multi-speaker environments remains a major challenge. While some approaches achieve strong performance when fine-tuned on specific domains, few systems generalize well across out-of-domain datasets. Our prior work, Diarization-Conditioned Whisper (DiCoW), leverages speaker diarization outputs as conditioning information and, with minimal fine-tuning, demonstrated strong multilingual and multi-domain performance. In this paper, we address a key limitation of DiCoW: ambiguity in Silence-Target-Non-target-Overlap (STNO) masks, where two or more fully overlapping speakers may have nearly identical conditioning despite differing transcriptions. We introduce SE-DiCoW (Self-Enrolled Diarization-Conditioned Whisper), which uses diarization output to locate an enrollment segment anywhere in the conversation where the target speaker is most active. This enrollment segment is used as fixed conditioning via cross-attention at each encoder layer. We further refine DiCoW with improved data segmentation, model initialization, and augmentation. Together, these advances yield substantial gains: SE-DiCoW reduces macro-averaged tcpWER by 52.4% relative to the original DiCoW on the EMMA MT-ASR benchmark.
Paper Structure (10 sections, 5 equations, 2 figures, 2 tables)

This paper contains 10 sections, 5 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. DiCoW: Diarization-Conditioned Whisper
  4. Self-Enrolled Diarization-Conditioned Whisper
  5. Additional DiCoW Improvements
  6. Experimental Setup
  7. Results
  8. Analysis of Self-Enrollment Mixture Composition
  9. Conclusion
  10. Acknowledgements

Figures (2)

  • Figure 1: Overview of the SE-DiCoW model architecture. Newly introduced parameter blocks are highlighted in red.
  • Figure 2: STNO ambiguity in highly overlapping speech regions. The STNO masks of James and Michael differ only at the positions highlighted in red, leaving a single (non-)target speaker frame for the model to exploit to track the target speaker.