Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Joint Learning Global-Local Speaker Classification to Enhance End-to-End Speaker Diarization and Recognition

Yuhang Dai, Haopeng Lin, Jiale Qian, Ruiqi Yan, Hao Meng, Hanke Xie, Hanlin Wen, Shunshun Yin, Ming Tao, Xie Chen, Lei Xie, Xinsheng Wang

Abstract

Large Audio-Language Models (LALMs) have demonstrated remarkable performance in end-to-end speaker diarization and recognition. However, their speaker discriminability remains limited due to the scarcity of large-scale conversational data and the absence of explicit speaker representation optimization. To address this, we propose GLSC-SDR, a paradigm that jointly trains speaker classification with diarization and recognition. We further introduce a Global-Local Speaker Classification strategy, which uses clustered speakers as global labels and re-encoded intra-cluster speakers as local labels. This hierarchical design enhances fine-grained speaker discrimination while preserving semantic transcription accuracy. Experiments on AliMeeting, AISHELL-4, and AMI-SDM demonstrate that GLSC-SDR achieves competitive or superior performance compared to simulation-based and multi-encoder approaches, without relying on large-scale real conversational data.

Joint Learning Global-Local Speaker Classification to Enhance End-to-End Speaker Diarization and Recognition

Abstract

Large Audio-Language Models (LALMs) have demonstrated remarkable performance in end-to-end speaker diarization and recognition. However, their speaker discriminability remains limited due to the scarcity of large-scale conversational data and the absence of explicit speaker representation optimization. To address this, we propose GLSC-SDR, a paradigm that jointly trains speaker classification with diarization and recognition. We further introduce a Global-Local Speaker Classification strategy, which uses clustered speakers as global labels and re-encoded intra-cluster speakers as local labels. This hierarchical design enhances fine-grained speaker discrimination while preserving semantic transcription accuracy. Experiments on AliMeeting, AISHELL-4, and AMI-SDM demonstrate that GLSC-SDR achieves competitive or superior performance compared to simulation-based and multi-encoder approaches, without relying on large-scale real conversational data.

Paper Structure

This paper contains 15 sections, 1 equation, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Problem Formulation
  4. GLSC
  5. GLSC Data Construction Pipeline
  6. Experiments
  7. Experimental Setup and Implementation Details
  8. Datasets
  9. Model Configuration and Training Details
  10. Evaluation Metrics
  11. Results and Analysis
  12. Main results
  13. Analysis of classification methods
  14. Impact of clustering number
  15. Conclusion

Figures (2)

  • Figure 1: Joint training Global-Local Speaker Classification and Speech Diarization and Recognition paradigm.
  • Figure 2: GLSC data construction pipeline.