Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning

TL;DR

This work analyzes catastrophic forgetting in transformer-based LLMs during sequential fine-tuning, addressing how learned knowledge deteriorates as new tasks are introduced. It builds a mechanistic framework identifying three interacting processes—gradient interference on attention, representational drift in intermediate layers, and loss-landscape flattening near prior task minima—and shows these effects are modulated by model scale and task similarity, with a strong empirical link to forgetting (e.g., $r = 0.87$). By demonstrating causal relationships via targeted interventions (attention freezing, representation realignment, curvature regularization), the study delivers actionable strategies to mitigate forgetting and informs future architecture and optimization designs for robust continual learning. The findings advance mechanistic interpretability of LLMs and offer practical guidance for deploying continually adapting models with reduced catastrophic forgetting, scalable across large parameter budgets.

Abstract

Large language models exhibit remarkable performance across diverse tasks through pre-training and fine-tuning paradigms. However, continual fine-tuning on sequential tasks induces catastrophic forgetting, where newly acquired knowledge interferes with previously learned capabilities. Despite widespread observations of this phenomenon, the mechanistic understanding remains limited. Here, we present a comprehensive mechanistic analysis of catastrophic forgetting in transformer-based LLMs during sequential fine-tuning. Through systematic experiments across multiple model scales (109B to 400B total parameters) and task sequences, we identify three primary mechanisms driving forgetting: gradient interference in attention weights, representational drift in intermediate layers, and loss landscape flattening. We demonstrate that forgetting severity correlates strongly with task similarity (Pearson r = 0.87) and gradient alignment metrics. Our analysis reveals that approximately 15 to 23 percent of attention heads undergo severe disruption during fine-tuning, with lower layers showing greater susceptibility. These findings establish mechanistic foundations for developing targeted mitigation strategies in continual learning systems.

Figures (16)

• Figure 1: Catastrophic forgetting across model scales and task sequences. a, Schematic of sequential fine-tuning paradigm showing four-task sequence with performance tracking on all tasks at each stage. b, Forgetting curves for Llama 4 Maverick on high-similarity, medium-similarity, and low-similarity task sequences. Points represent mean accuracy plus or minus standard deviation across five runs. c, Forgetting magnitude (final minus initial accuracy on Task 1) as a function of model architecture and task similarity. Larger models show reduced forgetting, but task similarity remains dominant predictor. d, Gradient similarity versus forgetting magnitude correlation across all sequences and models ($r = 0.87$).
• Figure 1: Additional model architectures and hyperparameter sensitivity analysis.
• Figure 2: Attention mechanism disruption drives early-stage forgetting. a, Heatmap showing percentage of severely disrupted attention heads (Euclidean distance greater than 2.5 sigma) across layers for 40-layer model during fine-tuning. Lower layers show higher disruption rates. b, Attention entropy changes for disrupted versus stable heads. Disrupted heads exhibit mean entropy increase of 2.1 bits. c, Component ablation experiment: forgetting magnitude when selectively freezing different architectural components. Freezing attention layers reduces forgetting by 64 percent. d, Performance recovery by ablating disrupted heads after fine-tuning. Removing top 20 percent disrupted heads restores 47 percent of lost performance while minimally affecting new task.
• Figure 2: Task-specific performance curves for all 24 individual tasks across all sequences.
• Figure 3: Representational drift in intermediate layers. a, CKA similarity between pre-fine-tuning and post-fine-tuning representations across all layers. Intermediate layers (12 to 24) show largest drift with CKA decreases of 0.32 to 0.47. b, Principal component rotation angles during fine-tuning for leading (explaining greater than 60 percent variance) versus higher-order components. Leading components rotate 35 to 52 degrees while higher-order components remain stable (8 to 15 degree rotation). c, Representation realignment intervention: applying learned affine transformations to restore pre-fine-tuning geometry recovers 38 percent of lost performance. d, Correlation between CKA drift magnitude and forgetting severity across layers.