Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning

Duygu Nur Yaldiz, Evangelia Spiliopoulou, Zheng Qi, Siddharth Varia, Srikanth Doss, Nikolaos Pappas

TL;DR

This work addresses the calibration problem in decision-making LLMs and compares two fine-tuning paradigms: SFT and RLVR. It shows RLVR enhances task accuracy but yields extreme overconfidence, while SFT improves calibration with smaller gains, revealing a calibration–classification trade-off. Through diagnostics, it finds that the final decision token acts as an extraction of the reasoning trace rather than a calibrated uncertainty signal, explaining RLVR’s failure to calibrate. The authors propose a calibration-aware reinforcement learning method that directly regulates the decision-token probability, preserving RLVR’s accuracy while mitigating overconfidence, including under distribution shift. The approach yields more reliable confidence estimates and robust calibration, offering a practical path toward decision-making LLMs that are both accurate and uncertainty-aware.

Abstract

Large language models (LLMs) are increasingly deployed in decision-making tasks, where not only accuracy but also reliable confidence estimates are essential. Well-calibrated confidence enables downstream systems to decide when to trust a model and when to defer to fallback mechanisms. In this work, we conduct a systematic study of calibration in two widely used fine-tuning paradigms: supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). We show that while RLVR improves task performance, it produces extremely overconfident models, whereas SFT yields substantially better calibration, even under distribution shift, though with smaller performance gains. Through targeted experiments, we diagnose RLVR's failure, showing that decision tokens act as extraction steps of the decision in reasoning traces and do not carry confidence information, which prevents reinforcement learning from surfacing calibrated alternatives. Based on this insight, we propose a calibration-aware reinforcement learning formulation that directly adjusts decision-token probabilities. Our method preserves RLVR's accuracy level while mitigating overconfidence, reducing ECE scores up to 9 points.

Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning

TL;DR

This work addresses the calibration problem in decision-making LLMs and compares two fine-tuning paradigms: SFT and RLVR. It shows RLVR enhances task accuracy but yields extreme overconfidence, while SFT improves calibration with smaller gains, revealing a calibration–classification trade-off. Through diagnostics, it finds that the final decision token acts as an extraction of the reasoning trace rather than a calibrated uncertainty signal, explaining RLVR’s failure to calibrate. The authors propose a calibration-aware reinforcement learning method that directly regulates the decision-token probability, preserving RLVR’s accuracy while mitigating overconfidence, including under distribution shift. The approach yields more reliable confidence estimates and robust calibration, offering a practical path toward decision-making LLMs that are both accurate and uncertainty-aware.

Abstract

Large language models (LLMs) are increasingly deployed in decision-making tasks, where not only accuracy but also reliable confidence estimates are essential. Well-calibrated confidence enables downstream systems to decide when to trust a model and when to defer to fallback mechanisms. In this work, we conduct a systematic study of calibration in two widely used fine-tuning paradigms: supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). We show that while RLVR improves task performance, it produces extremely overconfident models, whereas SFT yields substantially better calibration, even under distribution shift, though with smaller performance gains. Through targeted experiments, we diagnose RLVR's failure, showing that decision tokens act as extraction steps of the decision in reasoning traces and do not carry confidence information, which prevents reinforcement learning from surfacing calibrated alternatives. Based on this insight, we propose a calibration-aware reinforcement learning formulation that directly adjusts decision-token probabilities. Our method preserves RLVR's accuracy level while mitigating overconfidence, reducing ECE scores up to 9 points.
Paper Structure (43 sections, 6 equations, 4 figures, 6 tables)

This paper contains 43 sections, 6 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Decision-Making with LLMs
  4. Confidence Estimation
  5. Confidence Calibration
  6. Interpreting Calibration
  7. Goal: Accurate and Calibrated Model
  8. The Calibration–Classification Tradeoff in Fine-Tuning Paradigms
  9. Fine-Tuning Algorithms
  10. SFT
  11. RLVR
  12. Experimental Design
  13. Results and Discussion
  14. Diagnosing Calibration Failures in RLVR
  15. Why RLVR Cannot Achieve Calibration?
  16. ...and 28 more sections

Figures (4)

  • Figure 1: Reliability diagrams of Qwen3-1.7B model. Confidence distributions indicate that the Base and GRPO fine-tuned models exhibit extreme overconfidence, with most predictions assigned probabilities near 1. In contrast, SFT and our proposal produce a broader spread of confidence values and reliability diagrams that align more closely with the diagonal, indicating improved calibration.
  • Figure 2: Illustration of the swapping experiment.
  • Figure 3: Confidence distributions of flipped predictions in the reasoning-swap experiment.
  • Figure 4: Reliability diagrams and confidence distributions of all model-dataset pairs.