Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding

Mingyue Huo, Yiwen Shao, Yuheng Zhang

TL;DR

TagSpeech tackles the problem of joint multi-speaker ASR and diarization with explicit timestamps by introducing a dual-stream LLM-based framework that decouples semantic content and speaker identity. It uses a fine-tuned Semantic encoder via Serialized Output Training (SOT) and a lightweight Interleaved Numeric Time Anchor mechanism to achieve fine-grained temporal grounding without altering the LLM vocabulary. The method demonstrates substantial Diarization Error Rate (DER) improvements over strong end-to-end baselines on AMI and AliMeeting, while maintaining competitive cpWER/gWER and enabling data-efficient training by freezing the LLM and only training lightweight projectors. These innovations enable a true “who spoke what and when” capability in a single end-to-end model, with practical implications for meeting transcription and real-world multi-speaker understanding. Future work highlights longer-context, streaming scenarios and richer paralinguistic cues as potential enhancements.

Abstract

We present TagSpeech, a unified LLM-based framework that utilizes Temporal Anchor Grounding for joint multi-speaker ASR and diarization. The framework is built on two key designs: (1) decoupled semantic and speaker streams fine-tuned via Serialized Output Training (SOT) to learn turn-taking dynamics; and (2) an interleaved time anchor mechanism that not only supports fine-grained timestamp prediction but also acts as a synchronization signal between semantic understanding and speaker tracking. Compared to previous works that primarily focus on speaker-attributed ASR or implicit diarization, TagSpeech addresses the challenge of fine-grained speaker-content alignment and explicitly models "who spoke what and when" in an end-to-end manner. Experiments on AMI and AliMeeting benchmarks demonstrate that our method achieves consistent improvements in Diarization Error Rate (DER) over strong end-to-end baselines, including Qwen-Omni and Gemini, particularly in handling complex speech overlaps. Moreover, TagSpeech employs a parameter-efficient training paradigm in which the LLM backbone is frozen and only lightweight projectors are trained, resulting in strong performance with low computational cost.

TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding

TL;DR

TagSpeech tackles the problem of joint multi-speaker ASR and diarization with explicit timestamps by introducing a dual-stream LLM-based framework that decouples semantic content and speaker identity. It uses a fine-tuned Semantic encoder via Serialized Output Training (SOT) and a lightweight Interleaved Numeric Time Anchor mechanism to achieve fine-grained temporal grounding without altering the LLM vocabulary. The method demonstrates substantial Diarization Error Rate (DER) improvements over strong end-to-end baselines on AMI and AliMeeting, while maintaining competitive cpWER/gWER and enabling data-efficient training by freezing the LLM and only training lightweight projectors. These innovations enable a true “who spoke what and when” capability in a single end-to-end model, with practical implications for meeting transcription and real-world multi-speaker understanding. Future work highlights longer-context, streaming scenarios and richer paralinguistic cues as potential enhancements.

Abstract

We present TagSpeech, a unified LLM-based framework that utilizes Temporal Anchor Grounding for joint multi-speaker ASR and diarization. The framework is built on two key designs: (1) decoupled semantic and speaker streams fine-tuned via Serialized Output Training (SOT) to learn turn-taking dynamics; and (2) an interleaved time anchor mechanism that not only supports fine-grained timestamp prediction but also acts as a synchronization signal between semantic understanding and speaker tracking. Compared to previous works that primarily focus on speaker-attributed ASR or implicit diarization, TagSpeech addresses the challenge of fine-grained speaker-content alignment and explicitly models "who spoke what and when" in an end-to-end manner. Experiments on AMI and AliMeeting benchmarks demonstrate that our method achieves consistent improvements in Diarization Error Rate (DER) over strong end-to-end baselines, including Qwen-Omni and Gemini, particularly in handling complex speech overlaps. Moreover, TagSpeech employs a parameter-efficient training paradigm in which the LLM backbone is frozen and only lightweight projectors are trained, resulting in strong performance with low computational cost.
Paper Structure (50 sections, 7 equations, 5 figures, 7 tables)

This paper contains 50 sections, 7 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Task Definition and Disambiguation.
  3. Challenge.
  4. Contributions.
  5. Related Work
  6. Towards Unified Multi-speaker Modeling
  7. LALMs and Temporal Awareness
  8. Method
  9. Problem Formulation
  10. Dual-Stream Disentangled Encoders
  11. Semantic Encoder with SOT Fine-tuning.
  12. Speaker Encoder.
  13. Projector.
  14. Interleaved Numeric Time Anchor
  15. Structured Input--Output Alignment
  16. ...and 35 more sections

Figures (5)

  • Figure 1: Comparison between a conventional cascaded pipeline (top) and our end-to-end framework TagSpeech (bottom) for multi-speaker speech processing.
  • Figure 2: Overview of TagSpeech, an end-to-end multi-speaker ASR and diarization framework. Dual encoders are used, where the semantic encoder is fine-tuned via serialized output training (SOT, up-left). Speech features are interleaved with time anchors for time-awareness and dual-stream synchronization (red dashed lines). The framework generates a structured output, explicitly addressing who spoke what and when.
  • Figure 3: Impact of time anchor granularity reveals a trade-off between temporal precision and semantic coherence. Inserting at every 8 frames balances both cpWER and DER.
  • Figure 4: The miss rate accounts for the dominant variation in DER across anchor intervals, indicating that time anchors improve diarization primarily by reducing missed speaker activity.
  • Figure 5: Visualization of diarization timelines on a challenging sample with dense overlap. Baselines avoid overlapping timestamps, revealing a linearity bias.