Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Speaker-Reasoner: Scaling Interaction Turns and Reasoning Patterns for Timestamped Speaker-Attributed ASR

Zhennan Lin, Shuai Wang, Zhaokai Sun, Pengyuan Xie, Chuan Xie, Jie Liu, Qiang Zhang, Lei Xie

Abstract

Transcribing and understanding multi-speaker conversations requires speech recognition, speaker attribution, and timestamp localization. While speech LLMs excel at single-speaker tasks, multi-speaker scenarios remain challenging due to overlapping speech, backchannels, rapid turn-taking, and context window constraints. We propose Speaker-Reasoner, an end-to-end Speech LLM with agentic multi-turn temporal reasoning. Instead of single-pass inference, the model iteratively analyzes global audio structure, autonomously predicts temporal boundaries, and performs fine-grained segment analysis, jointly modeling speaker identity, gender, timestamps, and transcription. A speaker-aware cache further extends processing to audio exceeding the training context window. Trained with a three-stage progressive strategy, Speaker-Reasoner achieves consistent improvements over strong baselines on AliMeeting and AISHELL-4 datasets, particularly in handling overlapping speech and complex turn-taking.

Speaker-Reasoner: Scaling Interaction Turns and Reasoning Patterns for Timestamped Speaker-Attributed ASR

Abstract

Transcribing and understanding multi-speaker conversations requires speech recognition, speaker attribution, and timestamp localization. While speech LLMs excel at single-speaker tasks, multi-speaker scenarios remain challenging due to overlapping speech, backchannels, rapid turn-taking, and context window constraints. We propose Speaker-Reasoner, an end-to-end Speech LLM with agentic multi-turn temporal reasoning. Instead of single-pass inference, the model iteratively analyzes global audio structure, autonomously predicts temporal boundaries, and performs fine-grained segment analysis, jointly modeling speaker identity, gender, timestamps, and transcription. A speaker-aware cache further extends processing to audio exceeding the training context window. Trained with a three-stage progressive strategy, Speaker-Reasoner achieves consistent improvements over strong baselines on AliMeeting and AISHELL-4 datasets, particularly in handling overlapping speech and complex turn-taking.

Paper Structure

This paper contains 15 sections, 3 equations, 1 figure, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Method
  3. Model Architecture
  4. Iterative Temporal Interaction
  5. Speaker-aware Context Cache
  6. Training Objective and Curriculum
  7. Experiments
  8. Implementation Details
  9. Datasets
  10. Evaluation Metrics
  11. Performance Comparison with Baselines
  12. Ablation on Training Stages
  13. Performance on Long-form Audio
  14. Evaluation of Speaker Attributes
  15. Conclusion

Figures (1)

  • Figure 1: The overview of Speaker-Reasoner. The model employs an agentic multi-turn reasoning mechanism on the temporal axis, utilizing an indexing and slicing tool and a speaker-aware context cache to iteratively generate speaker identity, gender, timestamps, and transcription from raw multi-speaker audio.