Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaling Multi-Talker ASR with Speaker-Agnostic Activity Streams

Xiluo He, Alexander Polok, Jesús Villalba, Thomas Thebaud, Matthew Maciejewski

TL;DR

The paper tackles the high inference cost of multi-talker ASR by decoupling runtime from the number of active speakers using Heuristic Error Assignment Training (HEAT). HEAT merges speaker activities into two speaker-agnostic streams and conditions a target-speaker ASR backbone (DiCoW) via stream-specific activity cues, enabled by a Frame-Level Diarization Dependent Transformation. Through oracle and diarization-based experiments on AMI, ICSI, and SparseLibriMix, the approach achieves substantial runtime gains (RTFx improvements) while maintaining competitive time-constrained WER (tcORC-WER). The work demonstrates both the practical viability of two-stream conditioning and lays groundwork for end-to-end HEAT outputs and streaming backbones, with release-ready code for reproducibility.

Abstract

An increasingly common training paradigm for multi-talker automatic speech recognition (ASR) is to use speaker activity signals to adapt single-speaker ASR models for overlapping speech. Although effective, these systems require running the ASR model once per speaker, resulting in inference costs that scale with the number of speakers and limiting their practicality. In this work, we propose a method that decouples the inference cost of activity-conditioned ASR systems from the number of speakers by converting speaker-specific activity outputs into two speaker-agnostic streams. A central challenge is that naïvely merging speaker activities into streams significantly degrades recognition, since pretrained ASR models assume contiguous, single-speaker inputs. To address this, we design new heuristics aimed at preserving conversational continuity and maintaining compatibility with existing systems. We show that our approach is compatible with Diarization-Conditioned Whisper (DiCoW) to greatly reduce runtimes on the AMI and ICSI meeting datasets while retaining competitive performance.

Scaling Multi-Talker ASR with Speaker-Agnostic Activity Streams

TL;DR

The paper tackles the high inference cost of multi-talker ASR by decoupling runtime from the number of active speakers using Heuristic Error Assignment Training (HEAT). HEAT merges speaker activities into two speaker-agnostic streams and conditions a target-speaker ASR backbone (DiCoW) via stream-specific activity cues, enabled by a Frame-Level Diarization Dependent Transformation. Through oracle and diarization-based experiments on AMI, ICSI, and SparseLibriMix, the approach achieves substantial runtime gains (RTFx improvements) while maintaining competitive time-constrained WER (tcORC-WER). The work demonstrates both the practical viability of two-stream conditioning and lays groundwork for end-to-end HEAT outputs and streaming backbones, with release-ready code for reproducibility.

Abstract

An increasingly common training paradigm for multi-talker automatic speech recognition (ASR) is to use speaker activity signals to adapt single-speaker ASR models for overlapping speech. Although effective, these systems require running the ASR model once per speaker, resulting in inference costs that scale with the number of speakers and limiting their practicality. In this work, we propose a method that decouples the inference cost of activity-conditioned ASR systems from the number of speakers by converting speaker-specific activity outputs into two speaker-agnostic streams. A central challenge is that naïvely merging speaker activities into streams significantly degrades recognition, since pretrained ASR models assume contiguous, single-speaker inputs. To address this, we design new heuristics aimed at preserving conversational continuity and maintaining compatibility with existing systems. We show that our approach is compatible with Diarization-Conditioned Whisper (DiCoW) to greatly reduce runtimes on the AMI and ICSI meeting datasets while retaining competitive performance.

Paper Structure

This paper contains 11 sections, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Conditioning ASR Models Using Speaker Activity
  4. Heuristic Error Assignment Training
  5. Conditioning ASR Models on HEAT Reference Activities
  6. Data and Experimental Setup
  7. Data
  8. Training Details
  9. Evaluation
  10. Results and Discussion
  11. Conclusion

Figures (2)

  • Figure 1: Activity Conditioning with HEAT. The ASR model runs twice ---once per HEAT stream---regardless of the number of speakers.
  • Figure 2: An audio sample (a) with five utterances from three different speakers, as denoted by colors, split into two streams using the following heuristics with HEAT: (b) First-available, (c) Alternating, (d) Recency-continuity, and (e) Speaker-continuity.