DiscoPhon: Benchmarking the Unsupervised Discovery of Phoneme Inventories With Discrete Speech Units

Abstract

We introduce DiscoPhon, a multilingual benchmark for evaluating unsupervised phoneme discovery from discrete speech units. DiscoPhon covers 6 dev and 6 test languages, chosen to span a wide range of phonemic contrasts. Given only 10 hours of speech in a previously unseen language, systems must produce discrete units that are mapped to a predefined phoneme inventory, through either a many-to-one or a one-to-one assignment. The resulting sequences are evaluated for unit quality, recognition and segmentation. We provide four pretrained multilingual HuBERT and SpidR baselines, and show that phonemic information is available enough in current models for derived units to correlate well with phonemes, though with variations across languages.

Figures (2)

• Figure 1: Overview of the streams in the evaluation pipeline. The model maps a waveform to units $\bm{u}$, evaluated against the gold phones $\bm{p}$ via $\textnormal{PNMI}(\bm{p}, \bm{u})$. An assignment, here many-to-one, then assign units to phones $\bm{a}$, assessed for recognition with $\textnormal{PER}(\bm{p}, \bm{a})$ and segmentation with $R\textnormal{-value}(\bm{p}, \bm{a})$ and $F_1(\bm{p}, \bm{a})$.
• Figure 2: Analysis of recognition errors in a many-to-one evaluation.A: Breakdown of the phone error rate of SpidR VP-20 with 256 units into insertions, substitutions, and deletions for each language, comparing zero-shot and finetuned models. B: Substitution confusion matrix between phoneme class for the finetuned model, averaged across languages. Each row indicates, given a ground-truth class of phonemes, which predicted class it was replaced by (in $\%$).