Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unsupervised lexicon learning from speech is limited by representations rather than clustering

Danel Adendorff, Simon Malan, Herman Kamper

TL;DR

The paper interrogates whether unsupervised lexicon learning from speech is bottlenecked by representations or clustering. By using ground-truth word boundaries and a range of self-supervised speech features alongside multiple clustering strategies across English and Mandarin, it demonstrates that variability in representations across word instances is the primary limiter, not clustering. When representations are idealized, clustering can yield perfect lexicons, highlighting the need to advance SSL representations for progress in zero-resource word discovery. The work also shows language-specific SSL pre-training significantly boosts Mandarin lexicon quality, underlining the value of language-aligned representations for this task.

Abstract

Zero-resource word segmentation and clustering systems aim to tokenise speech into word-like units without access to text labels. Despite progress, the induced lexicons are still far from perfect. In an idealised setting with gold word boundaries, we ask whether performance is limited by the representation of word segments, or by the clustering methods that group them into word-like types. We combine a range of self-supervised speech features (continuous/discrete, frame/word-level) with different clustering methods (K-means, hierarchical, graph-based) on English and Mandarin data. The best system uses graph clustering with dynamic time warping on continuous features. Faster alternatives use graph clustering with cosine distance on averaged continuous features or edit distance on discrete unit sequences. Through controlled experiments that isolate either the representations or the clustering method, we demonstrate that representation variability across segments of the same word type -- rather than clustering -- is the primary factor limiting performance.

Unsupervised lexicon learning from speech is limited by representations rather than clustering

TL;DR

The paper interrogates whether unsupervised lexicon learning from speech is bottlenecked by representations or clustering. By using ground-truth word boundaries and a range of self-supervised speech features alongside multiple clustering strategies across English and Mandarin, it demonstrates that variability in representations across word instances is the primary limiter, not clustering. When representations are idealized, clustering can yield perfect lexicons, highlighting the need to advance SSL representations for progress in zero-resource word discovery. The work also shows language-specific SSL pre-training significantly boosts Mandarin lexicon quality, underlining the value of language-aligned representations for this task.

Abstract

Zero-resource word segmentation and clustering systems aim to tokenise speech into word-like units without access to text labels. Despite progress, the induced lexicons are still far from perfect. In an idealised setting with gold word boundaries, we ask whether performance is limited by the representation of word segments, or by the clustering methods that group them into word-like types. We combine a range of self-supervised speech features (continuous/discrete, frame/word-level) with different clustering methods (K-means, hierarchical, graph-based) on English and Mandarin data. The best system uses graph clustering with dynamic time warping on continuous features. Faster alternatives use graph clustering with cosine distance on averaged continuous features or edit distance on discrete unit sequences. Through controlled experiments that isolate either the representations or the clustering method, we demonstrate that representation variability across segments of the same word type -- rather than clustering -- is the primary factor limiting performance.

Paper Structure

This paper contains 13 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Representations
  4. Clustering Methods
  5. Experimental Setup
  6. Data and Evaluation
  7. Implementation: Representations and Clustering Methods
  8. Experimental Results
  9. A First Evaluation: Representations
  10. Evaluation on English: Representations with Clustering
  11. Evaluation on Mandarin
  12. Representations vs Clustering
  13. Conclusion

Figures (1)

  • Figure 1: We consider different systems of (b) speech representations and (c) clustering approaches in order to learn (d) a lexicon from (a) unlabelled speech. For this study, we assume that true word boundaries are known.