Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

USM-SCD: Multilingual Speaker Change Detection Based on Large Pretrained Foundation Models

Guanlong Zhao, Yongqiang Wang, Jason Pelecanos, Yu Zhang, Hank Liao, Yiling Huang, Han Lu, Quan Wang

TL;DR

USM-SCD tackles multilingual speaker change detection and simultaneous ASR across $96$ languages by adapting a large foundation model for joint SCD/ASR. It leverages two pretraining streams—BEST-RQ unsupervised pretraining and ASR pretraining—and fine-tunes with training targets that insert a speaker change token between transcripts, enabling joint SCD/ASR with a Conformer-CTC backbone. The method shows that only approximately $25\%$ of trainable parameters need updating to achieve strong performance, and it introduces a decoding-time posterior scaling factor $\lambda>1$ that elevates SCD F1 to about $75.3\%$ at negligible WER cost. Compared with a strong monolingual En-US baseline and public ASR baselines, USM-SCD yields up to a $21\%$ relative improvement in SCD F1 and supports fast inference on TPUs (about $60\times$ real-time), highlighting its practical impact for scalable multilingual SCD/ASR systems.

Abstract

We introduce a multilingual speaker change detection model (USM-SCD) that can simultaneously detect speaker turns and perform ASR for 96 languages. This model is adapted from a speech foundation model trained on a large quantity of supervised and unsupervised data, demonstrating the utility of fine-tuning from a large generic foundation model for a downstream task. We analyze the performance of this multilingual speaker change detection model through a series of ablation studies. We show that the USM-SCD model can achieve more than 75% average speaker change detection F1 score across a test set that consists of data from 96 languages. On American English, the USM-SCD model can achieve an 85.8% speaker change detection F1 score across various public and internal test sets, beating the previous monolingual baseline model by 21% relative. We also show that we only need to fine-tune one-quarter of the trainable model parameters to achieve the best model performance. The USM-SCD model exhibits state-of-the-art ASR quality compared with a strong public ASR baseline, making it suitable to handle both tasks with negligible additional computational cost.

USM-SCD: Multilingual Speaker Change Detection Based on Large Pretrained Foundation Models

TL;DR

USM-SCD tackles multilingual speaker change detection and simultaneous ASR across languages by adapting a large foundation model for joint SCD/ASR. It leverages two pretraining streams—BEST-RQ unsupervised pretraining and ASR pretraining—and fine-tunes with training targets that insert a speaker change token between transcripts, enabling joint SCD/ASR with a Conformer-CTC backbone. The method shows that only approximately of trainable parameters need updating to achieve strong performance, and it introduces a decoding-time posterior scaling factor that elevates SCD F1 to about at negligible WER cost. Compared with a strong monolingual En-US baseline and public ASR baselines, USM-SCD yields up to a relative improvement in SCD F1 and supports fast inference on TPUs (about real-time), highlighting its practical impact for scalable multilingual SCD/ASR systems.

Abstract

We introduce a multilingual speaker change detection model (USM-SCD) that can simultaneously detect speaker turns and perform ASR for 96 languages. This model is adapted from a speech foundation model trained on a large quantity of supervised and unsupervised data, demonstrating the utility of fine-tuning from a large generic foundation model for a downstream task. We analyze the performance of this multilingual speaker change detection model through a series of ablation studies. We show that the USM-SCD model can achieve more than 75% average speaker change detection F1 score across a test set that consists of data from 96 languages. On American English, the USM-SCD model can achieve an 85.8% speaker change detection F1 score across various public and internal test sets, beating the previous monolingual baseline model by 21% relative. We also show that we only need to fine-tune one-quarter of the trainable model parameters to achieve the best model performance. The USM-SCD model exhibits state-of-the-art ASR quality compared with a strong public ASR baseline, making it suitable to handle both tasks with negligible additional computational cost.
Paper Structure (21 sections, 2 figures, 4 tables)

This paper contains 21 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Backbone model
  4. Pretraining
  5. BEST-RQ pretraining
  6. ASR pretraining
  7. SCD fine-tuning
  8. Speaker change token posterior scaling
  9. SCD evaluation metrics
  10. Experimental setup
  11. Data
  12. Training
  13. Evaluation
  14. Modeling details
  15. Results
  16. ...and 6 more sections

Figures (2)

  • Figure 1: Illustration of the SCD scoring mechanism for computing the precision, recall, and F1. "Spk A-C" stands for speaker annotations on a conversational utterance. "Ref SC" is the reference speaker change intervals. "Hyp SC" is the predicted speaker change. "Score" shows the scoring decision of each prediction and reference.
  • Figure 2: SCD token <st> posterior probability scaling results on YT-96-Eval.