SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervision

TL;DR

This paper tackles learning linguistic representations directly from raw speech without transcription. It introduces SpidR, a one‑pass self‑supervised model that uses self‑distillation and per‑layer codebook targets to stabilize online clustering and produce phonetic‑accessible representations. SpidR outperforms HuBERT and DinoSR on zero‑shot spoken language modeling benchmarks (sWUGGY, sBLIMP, tSC) and achieves substantial pretraining speedups (under a day on 16 GPUs); unit quality ABX and PNMI strongly correlate with LM performance, validating these proxies. The work demonstrates efficient textless language modeling and broadens access to resource‑scarce languages, with open‑source release.

Abstract

The parallel advances in language modeling and speech representation learning have raised the prospect of learning language directly from speech without textual intermediates. This requires extracting semantic representations directly from speech. Our contributions are threefold. First, we introduce SpidR, a self-supervised speech representation model that efficiently learns representations with highly accessible phonetic information, which makes it particularly suited for textless spoken language modeling. It is trained on raw waveforms using a masked prediction objective combined with self-distillation and online clustering. The intermediate layers of the student model learn to predict assignments derived from the teacher's intermediate layers. This learning objective stabilizes the online clustering procedure compared to previous approaches, resulting in higher quality codebooks. SpidR outperforms wav2vec 2.0, HuBERT, WavLM, and DinoSR on downstream language modeling benchmarks (sWUGGY, sBLIMP, tSC). Second, we systematically evaluate across models and layers the correlation between speech unit quality (ABX, PNMI) and language modeling performance, validating these metrics as reliable proxies. Finally, SpidR significantly reduces pretraining time compared to HuBERT, requiring only one day of pretraining on 16 GPUs, instead of a week. This speedup is enabled by the pretraining method and an efficient codebase, which allows faster iteration and easier experimentation. We open-source the training code and model checkpoints at https://github.com/facebookresearch/spidr.

Figures (16)

• Figure 1: Architecture of SpidR. The downsampling module, a stack of convolutional layers, transforms the speech waveform into 20ms frames. The student and teacher are Transformers with $L=12$ layers. For every layer $k$ in the last $K=8$ ones, the student predicts---through a prediction head $\phi^k$---the nearest neighbor codebook assignment on the masked positions from the teacher at the same layer. The downsampling module, the student and the prediction heads are updated by gradient descent (in blue). The teacher is an exponential moving average (EMA) of the student, and the codebooks are updated with an EMA of the embeddings of the teacher (in gray).
• Figure 2: Codebook and prediction perplexities during training for SpidR and DinoSR on LibriSpeech dev-clean, with $K = 8$ codebooks. For each layer $k$, the codebook perplexity is computed over each batch with $\bm{p} = \bm{y}^k$ and then averaged across the dataset. The prediction perplexity uses $\bm{p} = \tilde{\bm{y}}^k$.
• Figure 3: Data scaling results for a 125M parameters OPT model trained on Libri-Light, with different discrete units encoders. Zero-shot accuracy in %, chance level 50%. The speech encoders have $V = 256$ units. The log-likelihoods are normalized by the number of tokens, except for WUGGY with text.
• Figure 4: Approximate pretraining time for various hardware configurations with constant total batch size.
• Figure 5: Learning rate schedule (left) and EMA decay schedule of the teacher for DinoSR and SpidR (right).