SpidR-Adapt: A Universal Speech Representation Model for Few-Shot Adaptation

TL;DR

SpidR-Adapt tackles the data-inefficiency of self-supervised speech representations by introducing MAdaPT, a bi-level meta-training protocol, and FOBLO, a first-order optimization heuristic tailored for fast adaptation to new languages with minimal unlabeled data. It stabilizes learning with interleaved supervision and a robust initialization, enabling rapid performance gains in phonemic discriminability and spoken-language modeling, even with as little as 10 minutes to 100 hours of target-language data. Across OoD languages, SpidR-Adapt achieves data-efficiency up to 100x compared to standard pre-training and approaches in-domain performance, while also excelling on phoneme discovery benchmarks. The approach is architecture-agnostic and the authors provide open-source training code and model checkpoints for reproducibility and broader adoption.

Abstract

Human infants, with only a few hundred hours of speech exposure, acquire basic units of new languages, highlighting a striking efficiency gap compared to the data-hungry self-supervised speech models. To address this gap, this paper introduces SpidR-Adapt for rapid adaptation to new languages using minimal unlabeled data. We cast such low-resource speech representation learning as a meta-learning problem and construct a multi-task adaptive pre-training (MAdaPT) protocol which formulates the adaptation process as a bi-level optimization framework. To enable scalable meta-training under this framework, we propose a novel heuristic solution, first-order bi-level optimization (FOBLO), avoiding heavy computation costs. Finally, we stabilize meta-training by using a robust initialization through interleaved supervision which alternates self-supervised and supervised objectives. Empirically, SpidR-Adapt achieves rapid gains in phonemic discriminability (ABX) and spoken language modeling (sWUGGY, sBLIMP, tSC), improving over in-domain language models after training on less than 1h of target-language audio, over $100\times$ more data-efficient than standard training. These findings highlight a practical, architecture-agnostic path toward biologically inspired, data-efficient representations. We open-source the training code and model checkpoints at https://github.com/facebookresearch/spidr-adapt.

Figures (4)

• Figure 1: Overview of SpidR-Adapt for few-shot speech adaptation. It consists of three main phrases: (1) meta-initialization performs multi-task pre-training with interleaved supervision, learning a robust initialization $\bm{\phi}_0$ from a mixture of source domains. (2) meta-training through MAdaPT-FOBLO optimizes this initialization for fast adaption to $\mathcal{D}_{\ell}$. Each worker conducts inner-loop adaptation with active forgetting (AF) on unlabeled data, followed by outer-loop updates that refines $\phi$ by minimizing the expected task loss on labeled data. (3) at meta-test time, the learned $\bm \phi^*$ is fast adapted to a new, unseen domain using only unlabelled data.
• Figure 2: Data-efficiency of SpidR-Adapt on new languages across different adaptation data scales. We report ABX scores (lower is better) averaged across three test languages (French, German, English) for two initialization strategies (a) self-supervision [SSL] and (b) interleaved-supervision [SSL/SL]. Each sub-figure compares our approach with the baselines: In-Domain Mono-Task-PT, the oracle method pertained on 6k hours of in-domain data and Multi-Task-PT, standard multi-task pretraining using [SSL] or [SSL/SL] regimes. By integrating the proposed solution, MAdaPT-FOBLO, with Multi-Task-PT as meta-initialization, we achieve highly efficient adaptation to new languages. For detailed results, refer to Appendix \ref{['app:detailed_abx']}.
• Figure 3: Learning rate scheduler for FOBLO. We use blue and orange to represent the learning rate for self-supervised inner-steps and supervised outer-steps, respectively. The overall training has 200,000 steps. The learning rate scheduler alternates between inner-loop and outer-loop steps within each episode, with resets every 2,000 steps. The inner-loop uses a constant rate after a warmup, while the outer-loop follows a tri-stage schedule.
• Figure 4: Layer-wise analysis on the model's discriminability over phonemes. We present the ABX scores averaged over the corresponding new languages, and across the two within- and across-speaker conditions: (a) $5$ development and (b) $3$ test languages. We report results for our proposed MAdaPT-FOBLO method with two types of meta-initialization, Multi-Task-PT[SSL] and Multi-Task-PT[SSL/SL]. The optimal layer for ABX performance remains consistent across both ABX conditions, but varies depending on the meta-initialization. Specifically, the optimal layer is $6$ for initialization (a) and $8$ for initialization (b), respectively.