Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERT

Cheol Jun Cho, Abdelrahman Mohamed, Shang-Wen Li, Alan W Black, Gopala K. Anumanchipalli

TL;DR

The paper addresses unsupervised discovery of syllabic structure in speech SSL. It introduces SD-HuBERT, which fine-tunes HuBERT with a sentence-level aggregator token and a self-distillation objective to produce sentence representations, leading to emergent syllabic segmentation without labels. It also proposes SSABX, a tuning-free benchmark for evaluating sentence-level discriminability of speech models. Empirical results show SD-HuBERT outperforms baselines in unsupervised syllable discovery and SSABX, suggesting that sentence-level objectives can yield data-driven syllabic units useful for spoken language modeling.

Abstract

Data-driven unit discovery in self-supervised learning (SSL) of speech has embarked on a new era of spoken language processing. Yet, the discovered units often remain in phonetic space and the units beyond phonemes are largely underexplored. Here, we demonstrate that a syllabic organization emerges in learning sentence-level representation of speech. In particular, we adopt "self-distillation" objective to fine-tune the pretrained HuBERT with an aggregator token that summarizes the entire sentence. Without any supervision, the resulting model draws definite boundaries in speech, and the representations across frames exhibit salient syllabic structures. We demonstrate that this emergent structure largely corresponds to the ground truth syllables. Furthermore, we propose a new benchmark task, Spoken Speech ABX, for evaluating sentence-level representation of speech. When compared to previous models, our model outperforms in both unsupervised syllable discovery and learning sentence-level representation. Together, we demonstrate that the self-distillation of HuBERT gives rise to syllabic organization without relying on external labels or modalities, and potentially provides novel data-driven units for spoken language modeling.

SD-HuBERT: Sentence-Level Self-Distillation Induces Syllabic Organization in HuBERT

TL;DR

The paper addresses unsupervised discovery of syllabic structure in speech SSL. It introduces SD-HuBERT, which fine-tunes HuBERT with a sentence-level aggregator token and a self-distillation objective to produce sentence representations, leading to emergent syllabic segmentation without labels. It also proposes SSABX, a tuning-free benchmark for evaluating sentence-level discriminability of speech models. Empirical results show SD-HuBERT outperforms baselines in unsupervised syllable discovery and SSABX, suggesting that sentence-level objectives can yield data-driven syllabic units useful for spoken language modeling.

Abstract

Data-driven unit discovery in self-supervised learning (SSL) of speech has embarked on a new era of spoken language processing. Yet, the discovered units often remain in phonetic space and the units beyond phonemes are largely underexplored. Here, we demonstrate that a syllabic organization emerges in learning sentence-level representation of speech. In particular, we adopt "self-distillation" objective to fine-tune the pretrained HuBERT with an aggregator token that summarizes the entire sentence. Without any supervision, the resulting model draws definite boundaries in speech, and the representations across frames exhibit salient syllabic structures. We demonstrate that this emergent structure largely corresponds to the ground truth syllables. Furthermore, we propose a new benchmark task, Spoken Speech ABX, for evaluating sentence-level representation of speech. When compared to previous models, our model outperforms in both unsupervised syllable discovery and learning sentence-level representation. Together, we demonstrate that the self-distillation of HuBERT gives rise to syllabic organization without relying on external labels or modalities, and potentially provides novel data-driven units for spoken language modeling.
Paper Structure (12 sections, 3 figures, 2 tables)

This paper contains 12 sections, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Sentence-Level Fine-Tuning of HuBERT
  5. Unsupervised Syllable Discovery
  6. Spoken Sentence ABX (SSABX)
  7. Results
  8. Evaluation on Syllable Boundaries and Clustering
  9. Evaluation on Sentence-level Speech Embedding
  10. Why does syllabic organization emerge in SD-HuBERT?
  11. Conclusion
  12. Acknowledgements

Figures (3)

  • Figure 1: Diagram of the model architecture and sentence-level self-distillation framework. An aggregator token (Agg) is inserted to summarize the entire input speech.
  • Figure 2: Frame similarity matrices for "intelligence in others" from the 9th layer of HuBERT before fine-tuning, and the 9th and 11th layers after fine-tuning. The similarity is measured by dot product. The white dotted lines denote the ground truth syllable boundaries and the red dotted lines are the predicted boundaries. The syllables become clearly visible in SD-HuBERT after self-distillation. The frames are knocked out in the 11th layer of SD-HuBERT, drawing definite boundaries.
  • Figure 3: Layer-wise analysis of articulatory correlation cho2023emaprobing (top) and SSABX performance (bottom) of HuBERT (blue) and SD-HuBERT (orange).