SPEAR: A Unified SSL Framework for Learning Speech and Audio Representations

TL;DR

SPEAR introduces a unified self-supervised framework that bridges speech and general audio representation learning by distilling knowledge from domain-specific SSL teachers into a single encoder. It uses multi-codebook vector quantisation to generate fine-grained MVQ tokens from continuous teacher representations and employs an asymmetric dual-domain pre-training objective alongside a token-mixing augmentation to handle complex sound scenes. The approach yields state-of-the-art results on the SUPERB benchmark and strong performance on HEAR, while enabling substantial cross-domain transfer and a versatile footing for both speech and audio tasks. The work demonstrates that combining KD with fine-grained token prediction, plus targeted data fusion strategies, can produce a robust, general-purpose acoustic representation model with practical impact for diverse audio-processing applications.

Abstract

Self-supervised learning (SSL) has significantly advanced acoustic representation learning. However, most existing models are optimised for either speech or audio event understanding, resulting in a persistent gap between these two domains. We address this gap with SPEAR (SPEech and Audio Representations), a self-supervised framework that distils complementary knowledge from a speech-focused SSL teacher and a general-audio SSL teacher into a single unified model. SPEAR applies multi-codebook vector quantisation to continuous teacher representations to produce fine-grained discrete tokens that capture both semantic and acoustic information. To effectively integrate these heterogeneous representations, SPEAR jointly predicts them given a masked input with an asymmetric pre-training loss. We further improve robustness in complex sound scenes through a novel token mixing mechanism. Extensive experiments demonstrate that SPEAR consistently outperforms existing unified speech and audio models. SPEAR establishes a new state-of-the-art on the SUPERB benchmark, surpassing WavLM Large on 12 of 15 tasks, while achieving competitive performance on the HEAR benchmark. These results position SPEAR as a versatile foundation for general-purpose speech and audio representation learning. The code and pre-trained models will be released.

Figures (4)

• Figure 1: The framework for dual-domain pre-training. Teacher models are frozen. For single-domain pre-training, only one teacher from the corresponding field is employed for generating pre-training targets. After pre-training, the components in the grey box are retained as the encoder.
• Figure 2: A token mixing example with $\beta=0.6$ and $\tau=2$.
• Figure 3: Effect of $\alpha$ on two downstream fine-tuning tasks. Left: WERs of test-other on LS-100 ASR fine-tuning task; Right: mAP on AudioSet evaluation set on AS-balanced fine-tuning task.
• Figure 4: Comparing the reconstructed embedding space obtained through different MVQ quantisation and k-means. The speaker embeddings of 10 speakers drawn from LibriSpeech dev-clean are visualized using UMAP on a 2D plane, with each colour representing a single speaker. (a): Reconstruction using all codebooks of MVQ; (b): Reconstruction using only the 12th codebook of MVQ; (c): Reconstruction using k-means centroids.