Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation

Brady Steele

TL;DR

It is shown that rank affects forgetting only when task subspaces are similar (low angle), while orthogonal methods like O-LoRA provide minimal benefit when natural orthogonality is already high, reconciles seemingly contradictory findings in the literature.

Abstract

Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for adapting large pre-trained models, yet its behavior under continual learning remains poorly understood. We present a geometric theory characterizing catastrophic forgetting in LoRA through the lens of gradient subspace interactions. Our central finding is that forgetting is governed by a simple geometric law: $\mathcal{F} = α(1 - \cos^2θ_{\min}) + β$, where $θ_{\min}$ is the minimum principal angle between task gradient subspaces. This formulation reveals an approximate rank-invariance property, at high subspace angles, forgetting becomes largely independent of the adapter rank (coefficient of variation $\approx 0.8\%$ in controlled synthetic settings; CV $\approx 10$-$19\%$ on real benchmarks, suggesting this is regime-dependent rather than absolute). We validate our theory on synthetic tasks ($r=0.994$ correlation), Split-CIFAR100 with ViT-LoRA, and sequential GLUE with RoBERTa-LoRA. Our analysis reconciles seemingly contradictory findings in the literature: we show that rank affects forgetting only when task subspaces are similar (low angle), while orthogonal methods like O-LoRA provide minimal benefit when natural orthogonality is already high. These insights provide principled guidance for continual learning with parameter-efficient fine-tuning.

Subspace Geometry Governs Catastrophic Forgetting in Low-Rank Adaptation

TL;DR

It is shown that rank affects forgetting only when task subspaces are similar (low angle), while orthogonal methods like O-LoRA provide minimal benefit when natural orthogonality is already high, reconciles seemingly contradictory findings in the literature.

Abstract

Low-Rank Adaptation (LoRA) has emerged as a parameter-efficient approach for adapting large pre-trained models, yet its behavior under continual learning remains poorly understood. We present a geometric theory characterizing catastrophic forgetting in LoRA through the lens of gradient subspace interactions. Our central finding is that forgetting is governed by a simple geometric law: , where is the minimum principal angle between task gradient subspaces. This formulation reveals an approximate rank-invariance property, at high subspace angles, forgetting becomes largely independent of the adapter rank (coefficient of variation in controlled synthetic settings; CV - on real benchmarks, suggesting this is regime-dependent rather than absolute). We validate our theory on synthetic tasks ( correlation), Split-CIFAR100 with ViT-LoRA, and sequential GLUE with RoBERTa-LoRA. Our analysis reconciles seemingly contradictory findings in the literature: we show that rank affects forgetting only when task subspaces are similar (low angle), while orthogonal methods like O-LoRA provide minimal benefit when natural orthogonality is already high. These insights provide principled guidance for continual learning with parameter-efficient fine-tuning.
Paper Structure (45 sections, 3 theorems, 16 equations, 5 figures, 6 tables)

This paper contains 45 sections, 3 theorems, 16 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Key Contributions.
  3. Clarification of novelty.
  4. Related Work
  5. Parameter-Efficient Fine-Tuning.
  6. Continual Learning and Forgetting.
  7. Gradient Subspace Methods.
  8. LoRA and Forgetting.
  9. Theoretical Framework
  10. Problem Setup
  11. Geometric Forgetting Bound
  12. Rank-Invariance Corollary
  13. Rank-Angle Interaction
  14. Experiments
  15. Experimental Setup
  16. ...and 30 more sections

Key Result

Theorem 1

Under the following assumptions: the forgetting on task $i$ after training on task $t > i$ satisfies: where $\theta_{\min}(i,t)$ is the minimum principal angle between $\mathcal{G}_i$ and $\mathcal{G}_t$, $\alpha = \eta L \|\Delta_t\|^2 / \mu$ is a scaling factor depending on learning rate $\eta$, smoothness $L$, update norm $\|\Delta_t\|$, and curvature $\mu$, and $\beta \geq 0$ is baseline for

Figures (5)

  • Figure 1: Conceptual illustration of the geometric forgetting theory. (a) Gradient subspaces for two sequential tasks with principal angle $\theta_{\min}$ between them. (b) The geometric forgetting law: the separation term $(1-\cos^2\theta_{\min})$ increases with principal angle, and empirically correlates with observed forgetting.
  • Figure 2: Validation of the geometric forgetting law. The interference term $(1-\cos^2\theta_{\min})$ strongly predicts forgetting on both synthetic tasks (circles, $r=0.994$) and CIFAR-100 (squares). The fitted line $\mathcal{F} = 1.93(1-\cos^2\theta) - 0.07$ achieves $R^2 = 0.987$.
  • Figure 3: Approximate rank-invariance validation. (a) Synthetic experiments show nearly identical forgetting across ranks 1--32 (CV = 0.84%). (b) Real benchmarks (CIFAR-100 and GLUE) show approximate rank-invariance with CV $<20\%$, consistent with our theoretical prediction in high-angle regimes.
  • Figure 4: Layer-wise analysis of interference-forgetting correlation. Six out of seven LoRA layers (blue) show positive correlation between $(1-\cos^2\theta)$ and forgetting, supporting local validity of the geometric theory. The aggregate correlation (dashed line) is $r=0.525$.
  • Figure 5: Comparison of orthogonal methods. Vanilla LoRA and O-LoRA achieve nearly identical forgetting ($p=0.73$, not significant) when natural task orthogonality is already high. Error bars show standard deviation across seeds.

Theorems & Definitions (7)

  • Definition 1: Gradient Subspace
  • Definition 2: Principal Angles
  • Theorem 1: Geometric Forgetting Bound (Empirically Parameterized)
  • proof : Proof Sketch
  • Corollary 2: Rank Invariance at High Angles
  • Proposition 3: Extended Rank-Angle Theory
  • proof