RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging

TL;DR

This work tackles catastrophic forgetting in continual, multi-domain lidar-style language-model settings by proposing RECALL, a data-free, representation-aware model merging framework. RECALL leverages layer-wise internal representations from a small set of typical samples to compute inter-model similarities via an $m$-sample $RBF$-based metric and derives per-layer merging weights to perform hierarchical, layer-wise parameter fusion. The method demonstrates strong knowledge retention and cross-domain generalization across five NLP tasks, both when merging a base model with a single expert and when fusing multiple expert models, including in sequential fine-tuning scenarios where prior tasks are preserved more reliably than with standard baselines. With no access to historical data, RECALL offers a scalable approach to evolve LLMs while minimizing interference between specialized and general knowledge, enabling robust multi-domain integration in continual learning settings.

Abstract

We unveil that internal representations in large language models (LLMs) serve as reliable proxies of learned knowledge, and propose RECALL, a novel representation-aware model merging framework for continual learning without access to historical data. RECALL computes inter-model similarity from layer-wise hidden representations over clustered typical samples, and performs adaptive, hierarchical parameter fusion to align knowledge across models. This design enables the preservation of domain-general features in shallow layers while allowing task-specific adaptation in deeper layers. Unlike prior methods that require task labels or incur performance trade-offs, RECALL achieves seamless multi-domain integration and strong resistance to catastrophic forgetting. Extensive experiments across five NLP tasks and multiple continual learning scenarios show that RECALL outperforms baselines in both knowledge retention and generalization, providing a scalable and data-free solution for evolving LLMs.

Figures (7)

• Figure 1: A taxonomy of previous approaches to catastrophic forgetting. Data-based methods (a) rely on stored samples from previous tasks, which are replayed alongside new data during fine-tuning. Model-based methods (b) mitigate forgetting by either constraining parameter updates or isolating task-specific knowledge. In (b), the left side illustrates regularization-based methods that optimize model parameters within the intersection of low-loss regions for both old and new tasks (e.g., Task A and Task B), instead of strictly minimizing the loss on the new task. This encourages a more stable update trajectory that retains previously learned knowledge while adapting to new tasks.
• Figure 2: Illustration of representation transformation across layers within a single LLM. The input progresses through a sequence of transformations, and the corresponding hidden states (shown for layers 0, 15, and 32) exhibit distinct structural patterns in the representation space, highlighting the non-uniform nature of internal dynamics.
• Figure 3: Visualization of representation drift between two models fine-tuned on different tasks (SST2 vs. RACE). Despite sharing the same input, their hidden states evolve along different trajectories and form distinct clustering patterns, especially in deeper layers.
• Figure 4: Illustration of RECALL, our proposed representation-aware model merging framework. The pipeline consists of four stages: (1) extract hidden states from typical samples using the newly fine-tuned model $M_N$, (2) compute the pairwise representational similarities across all models (including $M_N$), (3) derive layer-wise adaptive weights based on similarity scores via softmax, and (4) perform hierarchical parameter merging guided by the computed weights. This process enables effective knowledge fusion across models while preserving task-specific features.
• Figure 5: Performance curves on SST-2 during sequential fine-tuning with other two baselines.