Entropy-Adaptive Fine-Tuning: Resolving Confident Conflicts to Mitigate Forgetting

TL;DR

This work investigates catastrophic forgetting in Supervised Fine-Tuning (SFT) for domain adaptation and contrasts it with on-policy RL, identifying a distributional gap manifested as Confident Conflicts (low probability and low entropy) that drives destructive gradient updates. It introduces Entropy-Adaptive Fine-Tuning (EAFT), a soft token-entropy gating mechanism that down-weights conflicting samples while focusing learning on high-entropy, uncertain tokens, using a Top-$K$ entropy approximation for efficiency. Across math, medical, and agent domains, and over models from 4B to 32B parameters (Qwen, GLM), EAFT achieves a Pareto improvement: it matches or exceeds target-domain performance while significantly mitigating forgetting of general capabilities and preserving robust knowledge. The approach is domain-agnostic, computationally efficient, and offers a principled way to balance adaptation with retention in large language models, with limitations noted and avenues for uncertainty calibration and broader knowledge-editing contexts suggested.

Abstract

Supervised Fine-Tuning (SFT) is the standard paradigm for domain adaptation, yet it frequently incurs the cost of catastrophic forgetting. In sharp contrast, on-policy Reinforcement Learning (RL) effectively preserves general capabilities. We investigate this discrepancy and identify a fundamental distributional gap: while RL aligns with the model's internal belief, SFT forces the model to fit external supervision. This mismatch often manifests as "Confident Conflicts" tokens characterized by low probability but low entropy. In these instances, the model is highly confident in its own prediction but is forced to learn a divergent ground truth, triggering destructive gradient updates. To address this, we propose Entropy-Adaptive Fine-Tuning (EAFT). Unlike methods relying solely on prediction probability, EAFT utilizes token-level entropy as a gating mechanism to distinguish between epistemic uncertainty and knowledge conflict. This allows the model to learn from uncertain samples while suppressing gradients on conflicting data. Extensive experiments on Qwen and GLM series (ranging from 4B to 32B parameters) across mathematical, medical, and agentic domains confirm our hypothesis. EAFT consistently matches the downstream performance of standard SFT while significantly mitigating the degradation of general capabilities.

Figures (7)

• Figure 1: (a) Conceptual illustration. When SFT forces the model to override its strong priors (e.g., labeling a "ball" as a "truncated icosahedron"), it creates a Confident Conflict. Fitting these conflicts distorts the model's existing representations, leading to catastrophic forgetting. (b) Token-level entropy–probability landscape. Compared to on-policy rollouts (right), the SFT data (left) exhibits a prominent cluster of Low Entropy, Low Probability tokens.
• Figure 2: (a) Masking "Confident Conflict" tokens (the bottom 15% in both entropy and probability) effectively mitigates the general capability degradation observed in standard SFT. (b) Across Math, Medical, and Agent domains, EAFT matches SFT in target task improvements (upper bars) while significantly minimizing performance drops on general benchmarks (lower bars).
• Figure 3: Gradient Magnitude Landscape. Left: SFT exerts strong optimization pressure (dark purple) on Confident Conflicts in the bottom-left. Right: EAFT effectively suppresses these gradients (light yellow), protecting the model's existing representations.
• Figure 4: Training dynamics of token subgroups. EAFT matches SFT on high-entropy tokens while keeping losses stable on low-entropy conflicts, preventing over-optimization of conflicting priors. High and low entropy correspond to values $\ge 2.0$ and $\le 0.5$, respectively.
• Figure 5: Pareto trade-off analysis. Unlike Masked SFT (drop in target score) or Standard SFT (severe forgetting), EAFT variants consistently occupy the optimal top-right frontier. This confirms that soft entropy-gating effectively preserves general capabilities without compromising target domain adaptation.