Putting a Face to Forgetting: Continual Learning meets Mechanistic Interpretability

TL;DR

Catastrophic forgetting is traditionally assessed at end-task performance, but this work presents a mechanistic, feature-centric view that explains forgetting as geometric transformations of feature encodings. By formalizing rotations and scaling of feature vectors and their effects on allocated capacity and readout, the authors derive best- and worst-case forgetting scenarios in a tractable feature-reader model and validate them experimentally. The framework is scaled to practice via Crosscoders, demonstrated in a Vision Transformer trained on Split CIFAR-10, where fading and readout misalignment emerge as primary forgetting drivers, with depth further aggravating the effect. This mechanistic lens offers a concrete vocabulary and diagnostic tools for understanding and mitigating forgetting in real-world continual-learning systems.

Abstract

Catastrophic forgetting in continual learning is often measured at the performance or last-layer representation level, overlooking the underlying mechanisms. We introduce a mechanistic framework that offers a geometric interpretation of catastrophic forgetting as the result of transformations to the encoding of individual features. These transformations can lead to forgetting by reducing the allocated capacity of features (worse representation) and disrupting their readout by downstream computations. Analysis of a tractable model formalizes this view, allowing us to identify best- and worst-case scenarios. Through experiments on this model, we empirically test our formal analysis and highlight the detrimental effect of depth. Finally, we demonstrate how our framework can be used in the analysis of practical models through the use of Crosscoders. We present a case study of a Vision Transformer trained on sequential CIFAR-10. Our work provides a new, feature-centric vocabulary for continual learning.

Figures (13)

• Figure 1: Linear representation hypothesis. (1) A feature is a specific concept or pattern in the data (e.g., curvature) that is encoded by a linear direction in a layer's activation space. The direction and strength with which a feature is represented are captured by a feature vector. (2) These feature vectors form the basis for the layer’s representation: its activation in response to an input $\bm x$ is a linear combination of feature vectors $\bm{\phi}_i$ weighted by their activations $f_i(\bm{x})$. (3) Deeper layers encode increasingly abstract features.
• Figure 2: Effect of shared features on capacity and accuracy forgetting. Tasks with shared but misaligned features (full) show more forgetting (F-Accuracy) due to misalignment (F-Gamma) and capacity degradation (F-Capacity Norm and F-Norm) than tasks with disjoint sets of features (none).
• Figure 3: Impact of the number of probes on allocated capacity. Increasing the number of probes that read the activation space worsens both overlap (F-Capacity Norm) and fading (F-Norm).
• Figure 4: Effect of depth on allocated capacity. Depth worsens overlap (F-Capacity Norm) and causes severe fading (F-Norm) in tasks with shared but misaligned features (Full). The effect is milder in features with disjoint sets of active features (None).
• Figure 5: Crosscoder diagram: Representations $\bm a_{\ell}^{m_t}(\bm x)$ from multiple models are mapped to a single shared latent space $\bm f_{\text{enc}}(\bm x)$, which is then used by model-specific decoders to reconstruct the original activations.

Theorems & Definitions (15)

• Definition 3.1: Feature contribution $\beta_i$
• Definition 3.2: Probe sensitivity $\gamma_i$
• Lemma 3.3: Expected feature vector update under Task $B$ training
• proof
• Theorem 3.4: Exact loss-change after learning task $B$
• proof
• Proposition 2.1: Gradient load sharing between probe and features
• proof
• Corollary 2.2: Fixed-probe exaggerates forgetting
• proof