LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning
Liangzu Peng, Juan Elenter, Joshua Agterberg, Alejandro Ribeiro, René Vidal
TL;DR
LoRanPAC addresses the instability of continual learning with lifted, high-dimensional random features by continually truncating the smallest SVD factors of the lifted feature matrix $H_{1:t}$. This yields a numerically stable, over-parameterized minimum-norm least-squares solution with theoretical guarantees: small training and test errors provided a suitable fraction of SVD components is retained, governed by the eigengap $\gamma_t$ and accumulative truncation error $a_t$. The method, which combines pre-trained features with a shallow random-feature head, scales to hundreds of tasks and outperforms state-of-the-art CL baselines like RanPAC across multiple datasets, while maintaining stability under Inc-1 scenarios. The contribution demonstrates how principled numerical techniques (continual SVD truncation) can be integrated with theory-informed bounds to produce robust, scalable continual learning systems in practical settings.
Abstract
The goal of continual learning (CL) is to train a model that can solve multiple tasks presented sequentially. Recent CL approaches have achieved strong performance by leveraging large pre-trained models that generalize well to downstream tasks. However, such methods lack theoretical guarantees, making them prone to unexpected failures. Conversely, principled CL approaches often fail to achieve competitive performance. In this work, we aim to bridge this gap between theory and practice by designing a simple CL method that is theoretically sound and highly performant. Specifically, we lift pre-trained features into a higher dimensional space and formulate an over-parametrized minimum-norm least-squares problem. We find that the lifted features are highly ill-conditioned, potentially leading to large training errors (numerical instability) and increased generalization errors. We address these challenges by continually truncating the singular value decomposition of the lifted features. Our approach, termed LoRanPAC, is stable with respect to the choice of hyperparameters, can handle hundreds of tasks, and outperforms state-of-the-art CL methods on multiple datasets. Importantly, our method satisfies a recurrence relation throughout its continual learning process, which allows us to prove it maintains small training and test errors by appropriately truncating a fraction of SVD factors. This results in a stable continual learning method with strong empirical performance and theoretical guarantees. Code available: https://github.com/liangzu/loranpac.