Scalable Analytic Classifiers with Associative Drift Compensation for Class-Incremental Learning of Vision Transformers

TL;DR

This paper addresses the bottleneck of classifier reconstruction in class-incremental learning with Vision Transformers by introducing LR-RGDA, a scalable analytic classifier that decomposes the discriminant into a global affine term and a low-rank class-specific quadratic, reducing inference from $\mathcal{O}(Cd^2)$ to $\mathcal{O}(d^2 + Cdr)$. To combat representation drift from backbone updates, it proposes HopDC, a training-free drift compensator based on Modern Continuous Hopfield Networks that aligns historical statistics using unlabeled anchors with a theoretical error bound. The combination yields state-of-the-art performance across diverse ViT backbones and adaptation strategies, and demonstrates strong data-efficiency and scalability in large-scale CIL benchmarks. These contributions provide a practical, theoretically grounded solution for robust and efficient class-incremental learning in vision models. The work has potential impact on scalable continual learning in large-scale vision systems, enabling effective updates without retraining extensive classifiers.

Abstract

Class-incremental learning (CIL) with Vision Transformers (ViTs) faces a major computational bottleneck during the classifier reconstruction phase, where most existing methods rely on costly iterative stochastic gradient descent (SGD). We observe that analytic Regularized Gaussian Discriminant Analysis (RGDA) provides a Bayes-optimal alternative with accuracy comparable to SGD-based classifiers; however, its quadratic inference complexity limits its use in large-scale CIL scenarios. To overcome this, we propose Low-Rank Factorized RGDA (LR-RGDA), a scalable classifier that combines RGDA's expressivity with the efficiency of linear classifiers. By exploiting the low-rank structure of the covariance via the Woodbury matrix identity, LR-RGDA decomposes the discriminant function into a global affine term refined by a low-rank quadratic perturbation, reducing the inference complexity from $\mathcal{O}(Cd^2)$ to $\mathcal{O}(d^2 + Crd^2)$, where $C$ is the class number, $d$ the feature dimension, and $r \ll d$ the subspace rank. To mitigate representation drift caused by backbone updates, we further introduce Hopfield-based Distribution Compensator (HopDC), a training-free mechanism that uses modern continuous Hopfield Networks to recalibrate historical class statistics through associative memory dynamics on unlabeled anchors, accompanied by a theoretical bound on the estimation error. Extensive experiments on diverse CIL benchmarks demonstrate that our framework achieves state-of-the-art performance, providing a scalable solution for large-scale class-incremental learning with ViTs. Code: https://github.com/raoxuan98-hash/lr_rgda_hopdc.

Figures (14)

• Figure 1: RGDA vs. SGD-based classifiers performance on a 1001-class cross-domain dataset. By regularizing covariances as $\Sigma_c^{\rm reg} = \alpha_1 \Sigma_c + (1-\alpha_1)\Sigma_{\rm avg}$, RGDA interpolates between LDA ($\alpha_1=0$) and QDA ($\alpha_1=1$).
• Figure 2: Conceptual visualization of LR-RGDA’s discriminant function. (a) The global affine term $\mathcal{L}_c(\mathbf{x})$ provides linear decision boundaries using the average covariance. (b) Class-specific low-rank corrections $\mathcal{Q}_c(\mathbf{x})$ (visualized via $\exp(\mathcal{Q}_c(\mathbf{x}))$ for clarity) introduce nonlinear adjustments. (c) The final discriminant $g_c(\mathbf{x}) = \mathcal{L}_c(\mathbf{x}) + \mathcal{Q}_c(\mathbf{x})$ yields composite, nonlinear boundaries.
• Figure 3: Ablation study on the low-rank dimension $r$ of LR-RGDA on the cross-domain CIL dataset with different ViTs.
• Figure 4: Sensitivity analysis of $\alpha_1$ and $\alpha_2$ in regularized covariance matrices for SGD-based and LR-RGDA classifiers.
• Figure 5: Inference throughput comparison of different classifiers (log scale). We compare the samples-per-second throughput of LR-RGDA against linear baselines (LDA, SGD) and the full-rank QDA as the number of classes increases.

Theorems & Definitions (17)

• Lemma 1: Bayes Optimality of RGDA
• Remark 1
• Lemma 2: Woodbury Matrix Identity
• Proposition 1: Decomposition of LR-RGDA's discriminant function
• Remark 2
• Proposition 2: Error bound of HopDC's drift estimation
• Remark 3
• proof
• proof
• Proposition 3: General error bound of HopDC's drift estimation