Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Low-Rank Adaptation Reduces Catastrophic Forgetting in Sequential Transformer Encoder Fine-Tuning: Controlled Empirical Evidence and Frozen-Backbone Representation Probes

Ashish Pandey

Abstract

Sequential fine-tuning of pretrained language encoders often overwrites previously acquired capabilities, but the forgetting behavior of parameter-efficient updates remains under-characterized. We present a controlled empirical study of Low-Rank Adaptation (LoRA) in sequential transformer encoder fine-tuning with companion representation probes that test a frozen-backbone explanation of its robustness. In five full-validation BERT-base reruns on an RTE->MRPC->CoLA->SST-2 sequence, full fine-tuning yields 19.9%+/-4.8% average forgetting, whereas standard LoRA (r=8, query/value modules) yields 0.6%+/-1.4% (paired t-test, p=0.002, Cohen's d_s=3.12). Task-level analyses confirm this reduction is not merely an aggregate effect. Secondary experiments on RoBERTa-base show the same pattern, and the strongest EWC baseline remains at 15.5%+/-1.4% forgetting. A six-task extension reveals that low average forgetting can hide strong task-level heterogeneity. Fine-grained freezing ablations show a marked forgetting drop once frozen parameters exceed roughly 95%, with classifier-only and shallow-adapter baselines approaching LoRA. Companion task-similarity probes in GPT-2 and RoBERTa show the same directional story: frozen-backbone regimes preserve higher inter-task similarity than full fine-tuning, gradual unfreezing weakens stability, and full fine-tuning exhibits its clearest divergence at the final transformer layer. These results support a restrained mechanistic interpretation: LoRA helps largely because backbone freezing preserves a more stable shared feature scaffold. We position standard LoRA as both a strong empirical baseline for sequential encoder adaptation and a useful probe of how selective plasticity shapes interference in transformer continual learning.

Low-Rank Adaptation Reduces Catastrophic Forgetting in Sequential Transformer Encoder Fine-Tuning: Controlled Empirical Evidence and Frozen-Backbone Representation Probes

Abstract

Sequential fine-tuning of pretrained language encoders often overwrites previously acquired capabilities, but the forgetting behavior of parameter-efficient updates remains under-characterized. We present a controlled empirical study of Low-Rank Adaptation (LoRA) in sequential transformer encoder fine-tuning with companion representation probes that test a frozen-backbone explanation of its robustness. In five full-validation BERT-base reruns on an RTE->MRPC->CoLA->SST-2 sequence, full fine-tuning yields 19.9%+/-4.8% average forgetting, whereas standard LoRA (r=8, query/value modules) yields 0.6%+/-1.4% (paired t-test, p=0.002, Cohen's d_s=3.12). Task-level analyses confirm this reduction is not merely an aggregate effect. Secondary experiments on RoBERTa-base show the same pattern, and the strongest EWC baseline remains at 15.5%+/-1.4% forgetting. A six-task extension reveals that low average forgetting can hide strong task-level heterogeneity. Fine-grained freezing ablations show a marked forgetting drop once frozen parameters exceed roughly 95%, with classifier-only and shallow-adapter baselines approaching LoRA. Companion task-similarity probes in GPT-2 and RoBERTa show the same directional story: frozen-backbone regimes preserve higher inter-task similarity than full fine-tuning, gradual unfreezing weakens stability, and full fine-tuning exhibits its clearest divergence at the final transformer layer. These results support a restrained mechanistic interpretation: LoRA helps largely because backbone freezing preserves a more stable shared feature scaffold. We position standard LoRA as both a strong empirical baseline for sequential encoder adaptation and a useful probe of how selective plasticity shapes interference in transformer continual learning.

Paper Structure

This paper contains 30 sections, 4 equations, 11 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Catastrophic Forgetting and Continual Learning
  5. Parameter-Efficient Fine-Tuning
  6. PEFT for Continual Learning
  7. Task-Invariant Representations and Mechanistic Explanations
  8. Experimental Methodology
  9. Sequential Protocol and Forgetting Metric
  10. Evidence Tiers and Protocol Taxonomy
  11. Models, Hyperparameters, and Statistics
  12. Interpretive Scope
  13. Companion Mechanism Probes
  14. Main Results
  15. Confirmatory BERT-Base Result
  16. ...and 15 more sections

Figures (11)

  • Figure 1: Paper overview. Left: full fine-tuning leaves the entire backbone trainable and suffers high forgetting under unconstrained drift. Middle: LoRA freezes the backbone and routes task-specific adaptation through small low-rank updates, preserving a more stable shared scaffold. Right: the empirical freezing ablation shows a sharp forgetting drop once the frozen fraction crosses roughly 95%.
  • Figure 2: Seed-paired confirmatory BERT forgetting comparison from the audited full-validation reruns (seeds 45--49).
  • Figure 3: Task-level and stage-wise forgetting in the confirmatory BERT reruns (seeds 45--49). (a) LoRA reduces forgetting on every individual task. (b) The gap between methods widens progressively as more tasks are added.
  • Figure 4: Mean task-by-stage forgetting in the confirmatory BERT reruns. Each cell reports forgetting relative to that task's post-training peak, averaged across seeds 45--49.
  • Figure 5: Secondary cross-checks. Left: RoBERTa full fine-tuning versus LoRA. Right: EWC forgetting by $\lambda$ with confirmatory BERT reference lines.
  • ...and 6 more figures