Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Large-Scale Probing Analysis of Speaker-Specific Attributes in Self-Supervised Speech Representations

Aemon Yat Fei Chiu, Kei Ching Fung, Roger Tsz Yeung Li, Jingyu Li, Tan Lee

TL;DR

It is discovered that larger models unexpectedly recover speaker identity in their deep layers, and the intermediate representations of speech SSL models are found to capture dynamic prosody better than specialised speaker embeddings.

Abstract

Enhancing explainability in speech self-supervised learning (SSL) is important for developing reliable SSL-based speech processing systems. This study probes how speech SSL models encode speaker-specific information via a large-scale probing analysis of 11 models, decomposing identity into acoustic, prosodic, and paralinguistic attributes. The results confirm a general hierarchy wherein initial layers encode fundamental acoustics and middle layers synthesise abstract traits. Crucially, the consensus that final layers purely abstract linguistic content is challenged. It is discovered that larger models unexpectedly recover speaker identity in their deep layers. Furthermore, the intermediate representations of speech SSL models are found to capture dynamic prosody better than specialised speaker embeddings. These insights decode the complex internal mechanics of SSL models, providing guidelines for selecting interpretable and task-optimal representations.

A Large-Scale Probing Analysis of Speaker-Specific Attributes in Self-Supervised Speech Representations

TL;DR

It is discovered that larger models unexpectedly recover speaker identity in their deep layers, and the intermediate representations of speech SSL models are found to capture dynamic prosody better than specialised speaker embeddings.

Abstract

Enhancing explainability in speech self-supervised learning (SSL) is important for developing reliable SSL-based speech processing systems. This study probes how speech SSL models encode speaker-specific information via a large-scale probing analysis of 11 models, decomposing identity into acoustic, prosodic, and paralinguistic attributes. The results confirm a general hierarchy wherein initial layers encode fundamental acoustics and middle layers synthesise abstract traits. Crucially, the consensus that final layers purely abstract linguistic content is challenged. It is discovered that larger models unexpectedly recover speaker identity in their deep layers. Furthermore, the intermediate representations of speech SSL models are found to capture dynamic prosody better than specialised speaker embeddings. These insights decode the complex internal mechanics of SSL models, providing guidelines for selecting interpretable and task-optimal representations.
Paper Structure (18 sections, 4 figures, 2 tables)

This paper contains 18 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Probing single-vector deep speaker embeddings
  4. Layer-wise analysis of speech SSL models
  5. Experimental Settings
  6. Probing network
  7. Dataset
  8. Probed speaker-specific attributes
  9. Models under investigation
  10. Training settings
  11. Results and Analyses
  12. Hierarchy of attribute encoding
  13. Deconstruction of speaker identity in the representations
  14. Model-specific variations and the impact of scale
  15. Ablation study
  16. ...and 3 more sections

Figures (4)

  • Figure 1: A schematic diagram of typical speech SSL models wav2vec2hubertunispeech-satwavlm.
  • Figure 2: The overall design of the probing task for speech SSL representations.
  • Figure 3: Probing accuracies for the Base and Base Plus variants of speech SSL models on the test set.
  • Figure 4: Probing accuracies for the Large and XLarge variants of speech SSL models on the test set.