Knowledge is Not Enough: Injecting RL Skills for Continual Adaptation

TL;DR

This work tackles the knowledge cutoff challenge in LLMs by decoupling knowledge updates from reinforcement-learned reasoning skills. It introduces Parametric Skill Transfer (PaST), which extracts a domain-agnostic Skill Vector from a source domain's RL refinement and linearly injects it into a target model after lightweight SFT, avoiding expensive RL in the new domain. The main contributions are (i) empirical evidence that SFT and RL updates occupy nearly orthogonal subspaces, (ii) a two-stage transfer mechanism with iterative refinement, and (iii) strong empirical results on SQuAD, LooGLE, and ToolBench showing improved knowledge usage, long-context reasoning, and cross-domain tool-use performance. PaST demonstrates a scalable and efficient pathway for continual adaptation of LLMs across diverse knowledge domains, enabling robust reasoning with updated information while maintaining computational practicality.

Abstract

Large Language Models (LLMs) face the "knowledge cutoff" challenge, where their frozen parametric memory prevents direct internalization of new information. While Supervised Fine-Tuning (SFT) is commonly used to update model knowledge, it often updates factual content without reliably improving the model's ability to use the newly incorporated information for question answering or decision-making. Reinforcement Learning (RL) is essential for acquiring reasoning skills; however, its high computational cost makes it impractical for efficient online adaptation. We empirically observe that the parameter updates induced by SFT and RL are nearly orthogonal. Based on this observation, we propose Parametric Skill Transfer (PaST), a framework that supports modular skill transfer for efficient and effective knowledge adaptation. By extracting a domain-agnostic Skill Vector from a source domain, we can linearly inject knowledge manipulation skills into a target model after it has undergone lightweight SFT on new data. Experiments on knowledge-incorporation QA (SQuAD, LooGLE) and agentic tool-use benchmarks (ToolBench) demonstrate the effectiveness of our method. On SQuAD, PaST outperforms the state-of-the-art self-editing SFT baseline by up to 9.9 points. PaST further scales to long-context QA on LooGLE with an 8.0-point absolute accuracy gain, and improves zero-shot ToolBench success rates by +10.3 points on average with consistent gains across tool categories, indicating strong scalability and cross-domain transferability of the Skill Vector.

Figures (4)

• Figure 1: Overview of Parametric Skill Transfer (PaST). The motivation (left) illustrates how standard SFT fails to handle environmental errors, leading to hallucinations, while PaST enables robust execution by incorporating reasoning skills. Our approach is based on the empirical finding (top right) that parameter updates for knowledge ($\Delta W_{SFT}$) and skills ($\Delta W_{RL}$) are nearly orthogonal and reside in disentangled subspaces. PaST first extracts a domain-agnostic skill vector $v_{skill} = \theta_{S}^{rl} - \theta_{S}^{sft}$ from a source domain and then linearly injects it into a target model via $\theta_{final} = \theta_{T}^{sft} + \lambda \cdot v_{skill}$, enabling efficient and effective knowledge adaptation without requiring expensive reinforcement learning in the target domain.
• Figure 2: We visualize the layer-wise cosine similarity between the weight changes induced by SFT ($\Delta W_{\text{SFT}}$) and RL ($\Delta W_{\text{RL}}$) on the LooGLE task. The dominant near-zero values indicate that knowledge acquisition and skill learning modify the model parameters along nearly orthogonal subspaces.
• Figure 3: Zero-shot cross-domain generalization on StableToolBench. Success Rate across 20 RL-unseen target categories using a skill vector trained solely on Movies. PaST (dark blue) raises the average success rate by +10.3% over the Target SFT baseline (grey). All results are averaged over three independent runs.
• Figure 4: Control Experiment: Similarity between two SFT updates. We visualize the cosine similarity between parameter updates induced by two different rounds of SFT ($\Delta W_{\text{SFT21}}$ vs. $\Delta W_{\text{SFT2}}$) on LooGLE. Unlike the SFT-RL comparison, these updates show a clear positive correlation (red regions), indicating that knowledge injection tasks operate within a shared parameter subspace.