PACE: Pretrained Audio Continual Learning

TL;DR

This work tackles the fragility of pretrained audio models under distribution shifts by formulating audio continual learning (CL) and benchmarking it with six diverse tasks. The authors identify critical gaps with vision-based CL transfers to audio and propose PACE, a stage-wise framework that combines improved first-session adaptation, adaptive multi-session subspace-orthogonal PEFT, and boundary-aware perturbations to align representations across sessions. Across coarse and fine-grained benchmarks, PACE consistently outperforms state-of-the-art baselines and narrows the gap to joint training, while maintaining efficient training cost. The approach advances robust, scalable continual adaptation for audio tasks such as speech, environmental sounds, and music, with practical implications for real-world systems.

Abstract

Audio is a fundamental modality for analyzing speech, music, and environmental sounds. Although pretrained audio models have significantly advanced audio understanding, they remain fragile in real-world settings where data distributions shift over time. In this work, we present the first systematic benchmark for audio continual learning (CL) with pretrained models (PTMs), together with a comprehensive analysis of its unique challenges. Unlike in vision, where parameter-efficient fine-tuning (PEFT) has proven effective for CL, directly transferring such strategies to audio leads to poor performance. This stems from a fundamental property of audio backbones: they focus on low-level spectral details rather than structured semantics, causing severe upstream-downstream misalignment. Through extensive empirical study, we identify analytic classifiers with first-session adaptation (FSA) as a promising direction, but also reveal two major limitations: representation saturation in coarse-grained scenarios and representation drift in fine-grained scenarios. To address these challenges, we propose PACE, a novel method that enhances FSA via a regularized analytic classifier and enables multi-session adaptation through adaptive subspace-orthogonal PEFT for improved semantic alignment. In addition, we introduce spectrogram-based boundary-aware perturbations to mitigate representation overlap and improve stability. Experiments on six diverse audio CL benchmarks demonstrate that PACE substantially outperforms state-of-the-art baselines, marking an important step toward robust and scalable audio continual learning with PTMs.

Figures (10)

• Figure 1: Audio CL \ref{['fig:shift-a']} on SC2 suffers from clearly much stronger representation shifts between adjacent sessions than vision CL \ref{['fig:shift-b']} on ImageNet-R.
• Figure 2: Comparison of vision CL and audio CL. (a) and (b) present performance patterns on audio and image datasets in both coarse- and fine-grained settings. (c) shows that, despite strong first-task plasticity with PEFT-FT, large representation shifts lead to severe forgetting.
• Figure 3: Analysis of representation tuning in CL. (a) and (b) show RanPAC's first-session, future-session, and average performance across FSA epochs relative to joint training. (c) shows the gains from simply freezing shallow layers. (d) is the t-SNE visualization of VocalSet after FSA.
• Figure 4: The proposed PACE framework. Stage 1 performs first-session adaptation with LoRA, followed by analytic inference. Stage 2 introduces subspace-orthogonal PEFT via LoRA subtraction and gradient projection. Boundary-aware regularization involves adaptation in the first two stages. Stage 3 freezes the backbone for stable adaptation. : frozen; : tuning; $\color{red}\longrightarrow$: adaptation path.
• Figure 5: CKA kornblith2019similarity visualization shows representation changes of the first session classes across layers in CL.