Probability-Entropy Calibration: An Elastic Indicator for Adaptive Fine-tuning

TL;DR

Probability-Entropy Calibration introduces RankTuner, a rank-based token reweighting method that fuses ground-truth token probability $p_t$ with intrinsic token uncertainty $H_t$ through a Relative Rank Indicator. By bridging $(p_t,H_t)$ to rank quantities $R_t$ and $\mathbb{E}[R_t]$ via bounds and CMVT, it derives a practical Relative Scale $\mathcal{S}_t$ to modulate token losses, emphasizing under-learned, high-complexity tokens while dampening noise. Empirical results across mathematical reasoning benchmarks and out-of-distribution datasets show consistent gains over probability-only and entropy-only baselines, with ablations highlighting the essential roles of both signals. The approach offers a scalable, generalizable fine-tuning paradigm grounded in uncertainty-aware ranking, promising broader impact on reasoning and code-generation tasks.

Abstract

Token-level reweighting is a simple yet effective mechanism for controlling supervised fine-tuning, but common indicators are largely one-dimensional: the ground-truth probability reflects downstream alignment, while token entropy reflects intrinsic uncertainty induced by the pre-training prior. Ignoring entropy can misidentify noisy or easily replaceable tokens as learning-critical, while ignoring probability fails to reflect target-specific alignment. RankTuner introduces a probability--entropy calibration signal, the Relative Rank Indicator, which compares the rank of the ground-truth token with its expected rank under the prediction distribution. The inverse indicator is used as a token-wise Relative Scale to reweight the fine-tuning objective, focusing updates on truly under-learned tokens without over-penalizing intrinsically uncertain positions. Experiments on multiple backbones show consistent improvements on mathematical reasoning benchmarks, transfer gains on out-of-distribution reasoning, and pre code generation performance over probability-only or entropy-only reweighting baselines.

Figures (7)

• Figure 1: A joint view of token correctness and intrinsic uncertainty.(Left) Token-level visualization of three indicators: the ground-truth probability $p_t$, token entropy $H_t$, and our Relative Rank Indicator $I_t$ (Sec. \ref{['sec:methodology']}). Colors encode relative magnitude; arrows indicate the increasing direction. (Right) A schematic in the $(p_t,H_t)$ plane with four regimes (①--④) distinguished by $I_t$; the background color gradient encodes $I_t$ values; inset histograms show representative predictive distributions for typical tokens (e.g., "essentially", "is", "5", "6"); the dashed circle marks a Noise Region (⑤).
• Figure 2: Visualization and empirical validation of rank-based metrics on Qwen3-8B predicted chain-of-thought tokens from the Minerva Math dataset.(Left) 3D visualization of the Relative Rank Indicator $\mathcal{I}$ as a function of Rank $R$ and Expected Rank $\mathbb{E}[R]$. The indicator incentivizes accurate predictions (low $R$) specifically in difficult contexts (high $\mathbb{E}[R]$). (Middle) Rank $R$ vs. probability $p$, showing adherence to the upper bound $R \leq 1/p$ (Eq. \ref{['eq:rank_prob_bound']}). (Right) Expected rank $\mathbb{E}[R]$ vs. entropy $H$, demonstrating alignment with the lower bound in Eq. \ref{['eq:expected_rank_entropy_bound']}. Note that the subscript $t$ is omitted here as we represent aggregate statistics over all tokens.
• Figure 3: Ablations, baselines, and inference entropy on AIME24 and OlympiadBench. Left: We report Pass@1/Pass@16 and compare RankTuner with tuned Alpha Power ($\alpha{=}0.5$) and Entropy Reg ($\alpha{=}0.02$). Middle: We plot AIME24 Pass@k and further include two RankTuner ablations (w/o Prob, w/o Entropy), highlighting complementary roles of the probability- and entropy-aware terms. Right: We measure average inference entropy on AIME24 for Qwen2.5-Math-7B; the dashed line indicates the original (pre-finetuning) model and colors group methods by probability orientation (P-decay, P-neutral, P-boost).
• Figure 4: Two-dimensional view of token difficulty and correctness.(Left) Token-level visualization on a partial reasoning trace from Qwen3-8B on AIME24, reporting $p_t$, $H_t$, and the proposed unified indicator $I_t$ (formalized in Sec. \ref{['sec:methodology']}). The three rows correspond to $p_t$, $H_t$, and $I_t$, respectively. Colors encode relative magnitude (blue $\rightarrow$ larger, red $\rightarrow$ smaller); arrows indicate the ascending direction (note $H_t$ is reversed). $I_t$ is normalized around a neutral value of $1$.
• Figure 5: Error distributions for bound tightness on Qwen3-8B (Minerva Math, tokens 0--29).(Left) Distribution of $\frac{1}{R}-p$ (rank-based approximation of token probability). (Right) Distribution of $\frac{1}{s(H)}-\frac{1}{\mathbb{E}[R]}$, where $s(H)$ is the entropy-based lower bound in Eq. \ref{['eq:expected_rank_entropy_bound']} (so $1/s(H)$ is the corresponding theoretical bound on $1/\mathbb{E}[R]$).

Theorems & Definitions (7)

• Definition 4.1: Rank and Expected Rank
• Definition 4.2: Relative Rank Indicator
• Definition 4.3: Relative Competence Template
• Proposition 4.4: Rank--Probability Bound
• Proposition 4.5: Expected Rank--Entropy Bound
• Lemma 1.1: Rank--Probability Bound
• Lemma 1.2: Expected Rank--Entropy Bound