Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SCDNet: Self-supervised Learning Feature-based Speaker Change Detection

Yue Li, Xinsheng Wang, Li Zhang, Lei Xie

TL;DR

This work addresses Speaker Change Detection by leveraging self-supervised learning features within an end-to-end Conformer-based framework (SCDNet). It introduces a learnable layer-weighting fusion to identify the most informative SSL layer, and adds a contrastive loss to mitigate overfitting in frame-level SCD. Through extensive experiments on multiple real and artificial datasets, WavLm-based SSL representations—especially intermediate layers—consistently yield strong performance, with SCDNet achieving state-of-the-art results and the contrastive approach enhancing fine-tuning gains. The findings suggest SSL features, coupled with layer-aware fusion and contrastive learning, offer robust, scalable improvements for SCD with potential downstream benefits for ASR and captioning systems.

Abstract

Speaker Change Detection (SCD) is to identify boundaries among speakers in a conversation. Motivated by the success of fine-tuning wav2vec 2.0 models for the SCD task, a further investigation of self-supervised learning (SSL) features for SCD is conducted in this work. Specifically, an SCD model, named SCDNet, is proposed. With this model, various state-of-the-art SSL models, including Hubert, wav2vec 2.0, and WavLm are investigated. To discern the most potent layer of SSL models for SCD, a learnable weighting method is employed to analyze the effectiveness of intermediate representations. Additionally, a fine-tuning-based approach is also implemented to further compare the characteristics of SSL models in the SCD task. Furthermore, a contrastive learning method is proposed to mitigate the overfitting tendencies in the training of both the fine-tuning-based method and SCDNet. Experiments showcase the superiority of WavLm in the SCD task and also demonstrate the good design of SCDNet.

SCDNet: Self-supervised Learning Feature-based Speaker Change Detection

TL;DR

This work addresses Speaker Change Detection by leveraging self-supervised learning features within an end-to-end Conformer-based framework (SCDNet). It introduces a learnable layer-weighting fusion to identify the most informative SSL layer, and adds a contrastive loss to mitigate overfitting in frame-level SCD. Through extensive experiments on multiple real and artificial datasets, WavLm-based SSL representations—especially intermediate layers—consistently yield strong performance, with SCDNet achieving state-of-the-art results and the contrastive approach enhancing fine-tuning gains. The findings suggest SSL features, coupled with layer-aware fusion and contrastive learning, offer robust, scalable improvements for SCD with potential downstream benefits for ASR and captioning systems.

Abstract

Speaker Change Detection (SCD) is to identify boundaries among speakers in a conversation. Motivated by the success of fine-tuning wav2vec 2.0 models for the SCD task, a further investigation of self-supervised learning (SSL) features for SCD is conducted in this work. Specifically, an SCD model, named SCDNet, is proposed. With this model, various state-of-the-art SSL models, including Hubert, wav2vec 2.0, and WavLm are investigated. To discern the most potent layer of SSL models for SCD, a learnable weighting method is employed to analyze the effectiveness of intermediate representations. Additionally, a fine-tuning-based approach is also implemented to further compare the characteristics of SSL models in the SCD task. Furthermore, a contrastive learning method is proposed to mitigate the overfitting tendencies in the training of both the fine-tuning-based method and SCDNet. Experiments showcase the superiority of WavLm in the SCD task and also demonstrate the good design of SCDNet.
Paper Structure (15 sections, 6 equations, 2 figures, 6 tables)

This paper contains 15 sections, 6 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. Problem Formulation
  4. SCDNet
  5. SSL Features for SCDNet
  6. Fine-tuning SSL Models for SCD
  7. Experimental settings
  8. Dataset and Evaluation
  9. Implementation details
  10. Experimental Results
  11. SSL Representation Comparison
  12. Comparison with SOTA methods
  13. Ablation Study
  14. Experiments with artificial database
  15. Conclusions

Figures (2)

  • Figure 1: The architecture of the SCDNet (left) and sampling strategy for contrastive learning (right).
  • Figure 2: Weighting values of different layers in the weighted representation fusion method. Experiments are conducted using the AMI dataset.