Reinforcement Learning Finetunes Small Subnetworks in Large Language Models

TL;DR

Reinforcement learning finetuning of large language models unexpectedly updates only a sparse subnetwork (often 5–30%) while leaving most parameters essentially unchanged. Finetuning this emergent subnetwork in isolation reproduces the full RL-finetuned model’s performance and nearly matches its parameter values, suggesting an intrinsic, transferable structure within pretrained models. The sparsity is consistent across seeds, data orders, and RL algorithms, and is primarily driven by training on in-distribution data, with limited impact from KL regularization or gradient clipping. These findings point to opportunities for more efficient RL strategies that exploit update sparsity without sacrificing performance.

Abstract

Reinforcement learning (RL) yields substantial improvements in large language models (LLMs) downstream task performance and alignment with human values. Surprisingly, such large gains result from updating only a small subnetwork comprising just 5 percent to 30 percent of the parameters, with the rest effectively unchanged. We refer to this phenomenon as parameter update sparsity induced by RL. It is observed across all 7 widely used RL algorithms (e.g., PPO, GRPO, DPO) and all 10 LLMs from different families in our experiments. This sparsity is intrinsic and occurs without any explicit sparsity promoting regularizations or architectural constraints. Finetuning the subnetwork alone recovers the test accuracy, and, remarkably, produces a model nearly identical to the one obtained via full finetuning. The subnetworks from different random seeds, training data, and even RL algorithms show substantially greater overlap than expected by chance. Our analysis suggests that this sparsity is not due to updating only a subset of layers, instead, nearly all parameter matrices receive similarly sparse updates. Moreover, the updates to almost all parameter matrices are nearly full-rank, suggesting RL updates a small subset of parameters that nevertheless span almost the full subspaces that the parameter matrices can represent. We conjecture that the this update sparsity can be primarily attributed to training on data that is near the policy distribution, techniques that encourage the policy to remain close to the pretrained model, such as the KL regularization and gradient clipping, have limited impact.

Figures (7)

• Figure 1: Comparison in accumulated gradients in the SFT stage vs RL stage for popular released checkpoints. SFT stage has accumulated much denser updates, while RL is mostly sparse.
• Figure 2: In PRIME, 72% parameters are never updated, 8% have gradients canceling each other out, and 20% constitute the subnetwork that is consistently updated (§\ref{['sec:reason']})
• Figure 3: Layerwise and per-parameter-matrix update sparsity for DPO (left) and PRIME (right). All layers are similarly sparsely updated, with the only exception of the layer normalization layers, which receive little to no updates.
• Figure 4: Training loss for training DPO with subnetwork finetuning and full finetuning. Training the subnetwork in isolation consistently causes train loss to be lower.
• Figure 5: Update sparsity of intermediate checkpoints of a training run of PRIME. We observe that with more training the sparsity slowly decays.