CARE-RFT: Confidence-Anchored Reinforcement Finetuning for Reliable Reasoning in Large Language Models

TL;DR

CARE-RFT addresses the accuracy–calibration trade-off in reinforcement finetuning for large language models by introducing a confidence-anchored skew reverse KL penalty. The method preserves calibration while achieving reasoning gains comparable to unconstrained RFT, across GRPO, DAPO, and GSPO on Qwen2.5-3B/7B. The core idea is to bound upward deviations from the reference policy in uncertain regions and relax the anchor when the model is confident and rewards align, mitigating hallucination and miscalibration. Empirical results show improved trustworthiness and stability with maintained reasoning performance, suggesting this regularization design is key to reliable reasoning models.

Abstract

Reinforcement finetuning (RFT) has emerged as a powerful paradigm for unlocking reasoning capabilities in large language models. However, we identify a critical trade-off: while unconstrained RFT achieves strong reasoning performance, it severely compromises model trustworthiness by amplifying hallucination and worsening calibration; conversely, RKL-constrained RFT preserves trustworthiness but limits reasoning gains due to its unbounded penalty on exploratory deviations. To resolve this tension, we introduce CARE-RFT (Confidence-Anchored Regularized Reinforcement Finetuning), a novel method that replaces standard reverse KL regularization with a skew reverse KL divergence. CARE-RFT provides a confidence-sensitive penalty: it is bounded for confident, consistently rewarded explorations to enable reasoning, while unbounded elsewhere to preserve calibration. Extensive experiments across multiple model scales and RFT algorithms show that CARE-RFT achieves a superior balance, matching the reasoning performance of unconstrained RFT while recovering the trustworthiness and calibration of the base model. Our work establishes that careful, confidence-aware regularization is key to building both capable and trustworthy reasoning models.

Figures (5)

• Figure 1: CARE-RFT breaks the accuracy--calibration trade-off. Across GRPO, DAPO, and GSPO on Qwen2.5-3B, unconstrained RL boosts accuracy but destroys calibration on MATHhendrycks2021measuring and TruthfulQAlin2021truthfulqa, while RKL restores calibration at the cost of accuracy. CARE-RFT consistently moves each method toward the upper-right region—achieving strong reasoning gains and stable factual reliability.
• Figure 2: ECE plot comparing base model with its +Reward and -Reward Update checkpoints on TruthfulQA. Each plot visualizes the relationship between model confidence $\text{conf}(B_m)$—estimated via sampling and majority voting—and the actual correctness probability $\text{acc}(B_m)$. Models closer to the diagonal with lower Expected Calibration Error (ECE) are better calibrated. A darker color means more responses are concentrated in this confidence interval.
• Figure 3: Penalty landscapes for reverse KL ($\alpha=0$, left) and skew reverse KL ($\alpha=0.8$, right). Standard RKL imposes unbounded penalties whenever the reference strongly disfavors, while skew RKL introduces a one-sided bound: upward deviations are capped by a finite penalty, enabling stable exploration, whereas downward deviations remain strongly penalized to preserve calibration.
• Figure 4: Gradient coefficient distribution for RKL and SRKL across different skew values $\alpha$
• Figure 5: Token-level entropy during training of GRPO variants on Qwen2.5-3B. Unconstrained RFT leads to entropy collapse, causing overconfidence and high ECE. RKL prevents collapse but limits performance gains. CARE-RFT allows for controlled entropy reduction, achieving a balance that explains its superior calibration-performance trade-off.

Theorems & Definitions (1)

• proof