Ability Transfer and Recovery via Modularized Parameters Localization

TL;DR

The paper addresses how specialized abilities in LLMs are stored and transferred, revealing that ability-related activations concentrate in a small subset of channels and remain disentangled after fine-tuning. It introduces ACT, a method that identifies activation-difference channels via cross-model comparisons, transfers only those channels through masked task-vector updates, and applies lightweight post-transfer fine-tuning to stabilize integration. Empirically, ACT recovers forgotten multilingual mathematics and science abilities with minimal parameter changes and enables clean merging of multiple abilities with minimal interference, outperforming several baselines. The work demonstrates a practical, parameter-efficient path to modularity-driven ability transfer, with implications for continual learning and multi-ability deployment in multilingual, domain-specific LLMs.

Abstract

Large language models can be continually pre-trained or fine-tuned to improve performance in specific domains, languages, or skills, but this specialization often degrades other capabilities and may cause catastrophic forgetting. We investigate how abilities are distributed within LLM parameters by analyzing module activations under domain- and language-specific inputs for closely related models. Across layers and modules, we find that ability-related activations are highly concentrated in a small set of channels (typically <5\%), and these channels are largely disentangled with good sufficiency and stability. Building on these observations, we propose ACT (Activation-Guided Channel-wise Ability Transfer), which localizes ability-relevant channels via activation differences and selectively transfers only the corresponding parameters, followed by lightweight fine-tuning for compatibility. Experiments on multilingual mathematical and scientific reasoning show that ACT can recover forgotten abilities while preserving retained skills. It can also merge multiple specialized models to integrate several abilities into a single model with minimal interference. Our code and data will be publicly released.

Figures (21)

• Figure 1: Complementary cumulative distribution function (CCDF), i.e., the fraction of output channels whose averaged activation difference exceeds a given threshold. Across all views, activation differences exhibit highly skewed, heavy-tailed distributions, with large deviations concentrated in a small subset of channels.
• Figure 2: Cross-pair consistency of ability-specific channels. Left: Qwen2.5-7B overlap between top-1% activation channels from (Math-Instruct vs. Instruct) and (Coder-Instruct vs. Instruct); Right: LLaMA-2-13B overlap from (Tulu-2-DPO vs. LLaMA-2) and (WizardLM vs. LLaMA-2).
• Figure 3: Illustration of ACT. $W_{\text{base}}$ denotes a base model, while $W_1$ and $W_2$ denote two ability models.
• Figure 4: Performance comparison of transferring from Qwen-2.5-7B-Instruct to Qwen-2.5-Math-7B-Instruct. Top row: Comparison across different transfer ratios with $\lambda=0.5$. Bottom row: Comparison using 1% transfer ratio with Task Arithmetic across different $\lambda$.
• Figure 5: Qwen2.5-Math-7B-Instruct vs. Qwen2.5-7B-Instruct. CCDFs of channel-wise activation differences across 11 languages for math (solid) and science (dashed). Large activation differences concentrate in a small fraction of channels across all language-domain combinations.