Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Supervised Syllable Discovery Based on Speaker-Disentangled HuBERT

Ryota Komatsu, Takahiro Shinozaki

TL;DR

Experimental results show that the proposed speech-only self-supervised fine-tuning approach surpasses the current state-of-the-art method in most syllable segmentation and syllabic unit quality metrics on Librispeech, underscoring its effectiveness in promoting syllabic organization within speech-only models.

Abstract

Self-supervised speech representation learning has become essential for extracting meaningful features from untranscribed audio. Recent advances highlight the potential of deriving discrete symbols from the features correlated with linguistic units, which enables text-less training across diverse tasks. In particular, sentence-level Self-Distillation of the pretrained HuBERT (SD-HuBERT) induces syllabic structures within latent speech frame representations extracted from an intermediate Transformer layer. In SD-HuBERT, sentence-level representation is accumulated from speech frame features through self-attention layers using a special CLS token. However, we observe that the information aggregated in the CLS token correlates more with speaker identity than with linguistic content. To address this, we propose a speech-only self-supervised fine-tuning approach that separates syllabic units from speaker information. Our method introduces speaker perturbation as data augmentation and adopts a frame-level training objective to prevent the CLS token from aggregating paralinguistic information. Experimental results show that our approach surpasses the current state-of-the-art method in most syllable segmentation and syllabic unit quality metrics on Librispeech, underscoring its effectiveness in promoting syllabic organization within speech-only models.

Self-Supervised Syllable Discovery Based on Speaker-Disentangled HuBERT

TL;DR

Experimental results show that the proposed speech-only self-supervised fine-tuning approach surpasses the current state-of-the-art method in most syllable segmentation and syllabic unit quality metrics on Librispeech, underscoring its effectiveness in promoting syllabic organization within speech-only models.

Abstract

Self-supervised speech representation learning has become essential for extracting meaningful features from untranscribed audio. Recent advances highlight the potential of deriving discrete symbols from the features correlated with linguistic units, which enables text-less training across diverse tasks. In particular, sentence-level Self-Distillation of the pretrained HuBERT (SD-HuBERT) induces syllabic structures within latent speech frame representations extracted from an intermediate Transformer layer. In SD-HuBERT, sentence-level representation is accumulated from speech frame features through self-attention layers using a special CLS token. However, we observe that the information aggregated in the CLS token correlates more with speaker identity than with linguistic content. To address this, we propose a speech-only self-supervised fine-tuning approach that separates syllabic units from speaker information. Our method introduces speaker perturbation as data augmentation and adopts a frame-level training objective to prevent the CLS token from aggregating paralinguistic information. Experimental results show that our approach surpasses the current state-of-the-art method in most syllable segmentation and syllabic unit quality metrics on Librispeech, underscoring its effectiveness in promoting syllabic organization within speech-only models.
Paper Structure (16 sections, 2 equations, 4 figures, 2 tables)

This paper contains 16 sections, 2 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Baseline method
  4. Analysis
  5. Proposed Method
  6. Self-supervised fine-tuning
  7. Unit segmentation
  8. EXPERIMENTAL SETUP
  9. Evaluation metrics
  10. Dataset
  11. Implementation details
  12. RESULTS
  13. Syllable segmentation and discovered unit quality
  14. Effect of speaker disentanglement
  15. What is essential for syllabic organization?
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Self-similarity matrices of latent speech frame representations from the $l$-th Transformer layer. Red lines are ground truth syllable boundaries. The speech sample is "SURELY WE CAN SUBMIT WITH GOOD GRACE".
  • Figure 2: The proposed framework.
  • Figure 3: Overview of syllabic unit discovery.
  • Figure 4: Layer-wise analysis of syllable segmentation scores and syllabic unit quality using our proposed method.