Parabolic Continual Learning
Haoming Yang, Ali Hasan, Vahid Tarokh
TL;DR
The work addresses continual learning under evolving data by enforcing a parabolic PDE constraint on the expected loss profile $u(x,t)$ with a memory-buffer boundary. The proposed Parabolic Continual Learner (PCL) combines diffusion-driven interior dynamics $\partial_t u = \sigma \nabla^2 u + \ell_{f_\theta}(x)$ (with optional first-order drift) and a boundary condition $u|_{\mathcal{M}^{(\epsilon)}} = \ell_{f_\theta}$, enabling tractable bounds on forgetting and generalization via the Feynman-Kac representation for scalable computation. Empirically, PCL achieves competitive or superior final accuracy and robustness across online class-incremental benchmarks, corrupted labels, and imbalanced data, while offering favorable efficiency compared to some replay-based baselines. The framework provides a unified PDE-based perspective that links regularization and memory replay through boundary-driven loss evolution, with potential extensions to broader modalities and control-theoretic connections.
Abstract
Regularizing continual learning techniques is important for anticipating algorithmic behavior under new realizations of data. We introduce a new approach to continual learning by imposing the properties of a parabolic partial differential equation (PDE) to regularize the expected behavior of the loss over time. This class of parabolic PDEs has a number of favorable properties that allow us to analyze the error incurred through forgetting and the error induced through generalization. Specifically, we do this through imposing boundary conditions where the boundary is given by a memory buffer. By using the memory buffer as a boundary, we can enforce long term dependencies by bounding the expected error by the boundary loss. Finally, we illustrate the empirical performance of the method on a series of continual learning tasks.