Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning

Jiayun Wu, Jiashuo Liu, Zhiyuan Zeng, Tianyang Zhan, Tianle Cai, Wenhao Huang

TL;DR

<3-5 sentence high-level summary>This work addresses the persistent hallucination problem in LLMs by reframing model output as a calibrated decision, where abstention and claim-level uncertainty signaling are learned through behavioral calibration. It introduces three strategies—Explicit Risk Thresholding, Verbalized Confidence, and Critic Value—with Verbalized Confidence and two prior-distribution-based rewards (Uniform and Beta) showing strong calibration and uncertainty discrimination, often exceeding much larger frontier models in uncertainty metrics. The experiments on Qwen3-4B-Instruct demonstrate that smaller models can achieve state-of-the-art calibration (e.g., SNR gains on BeyondAIME and zero-shot calibration on SimpleQA) while maintaining competitive or improved uncertainty handling. The results highlight a transferable meta-skill: calibration of epistemic certainty can be learned and deployed to improve safety and reliability without requiring maximal raw accuracy.

Abstract

LLM deployment in critical domains is currently impeded by persistent hallucinations--generating plausible but factually incorrect assertions. While scaling laws drove significant improvements in general capabilities, theoretical frameworks suggest hallucination is not merely stochastic error but a predictable statistical consequence of training objectives prioritizing mimicking data distribution over epistemic honesty. Standard RLVR paradigms, utilizing binary reward signals, inadvertently incentivize models as good test-takers rather than honest communicators, encouraging guessing whenever correctness probability exceeds zero. This paper presents an exhaustive investigation into behavioral calibration, which incentivizes models to stochastically admit uncertainty by abstaining when not confident, aligning model behavior with accuracy. Synthesizing recent advances, we propose and evaluate training interventions optimizing strictly proper scoring rules for models to output a calibrated probability of correctness. Our methods enable models to either abstain from producing a complete response or flag individual claims where uncertainty remains. Utilizing Qwen3-4B-Instruct, empirical analysis reveals behavior-calibrated reinforcement learning allows smaller models to surpass frontier models in uncertainty quantification--a transferable meta-skill decouplable from raw predictive accuracy. Trained on math reasoning tasks, our model's log-scale Accuracy-to-Hallucination Ratio gain (0.806) exceeds GPT-5's (0.207) in a challenging in-domain evaluation (BeyondAIME). Moreover, in cross-domain factual QA (SimpleQA), our 4B LLM achieves zero-shot calibration error on par with frontier models including Grok-4 and Gemini-2.5-Pro, even though its factual accuracy is much lower.

Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning

TL;DR

<3-5 sentence high-level summary>This work addresses the persistent hallucination problem in LLMs by reframing model output as a calibrated decision, where abstention and claim-level uncertainty signaling are learned through behavioral calibration. It introduces three strategies—Explicit Risk Thresholding, Verbalized Confidence, and Critic Value—with Verbalized Confidence and two prior-distribution-based rewards (Uniform and Beta) showing strong calibration and uncertainty discrimination, often exceeding much larger frontier models in uncertainty metrics. The experiments on Qwen3-4B-Instruct demonstrate that smaller models can achieve state-of-the-art calibration (e.g., SNR gains on BeyondAIME and zero-shot calibration on SimpleQA) while maintaining competitive or improved uncertainty handling. The results highlight a transferable meta-skill: calibration of epistemic certainty can be learned and deployed to improve safety and reliability without requiring maximal raw accuracy.

Abstract

LLM deployment in critical domains is currently impeded by persistent hallucinations--generating plausible but factually incorrect assertions. While scaling laws drove significant improvements in general capabilities, theoretical frameworks suggest hallucination is not merely stochastic error but a predictable statistical consequence of training objectives prioritizing mimicking data distribution over epistemic honesty. Standard RLVR paradigms, utilizing binary reward signals, inadvertently incentivize models as good test-takers rather than honest communicators, encouraging guessing whenever correctness probability exceeds zero. This paper presents an exhaustive investigation into behavioral calibration, which incentivizes models to stochastically admit uncertainty by abstaining when not confident, aligning model behavior with accuracy. Synthesizing recent advances, we propose and evaluate training interventions optimizing strictly proper scoring rules for models to output a calibrated probability of correctness. Our methods enable models to either abstain from producing a complete response or flag individual claims where uncertainty remains. Utilizing Qwen3-4B-Instruct, empirical analysis reveals behavior-calibrated reinforcement learning allows smaller models to surpass frontier models in uncertainty quantification--a transferable meta-skill decouplable from raw predictive accuracy. Trained on math reasoning tasks, our model's log-scale Accuracy-to-Hallucination Ratio gain (0.806) exceeds GPT-5's (0.207) in a challenging in-domain evaluation (BeyondAIME). Moreover, in cross-domain factual QA (SimpleQA), our 4B LLM achieves zero-shot calibration error on par with frontier models including Grok-4 and Gemini-2.5-Pro, even though its factual accuracy is much lower.
Paper Structure (32 sections, 6 equations, 8 figures, 4 tables)

This paper contains 32 sections, 6 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Theoretical Investigations of Hallucination.
  4. Hallucination Detection.
  5. Abstention and Self-Verification.
  6. Verbalized Confidence.
  7. Test-Time Scaling with Confidence.
  8. Comparison with Related Work.
  9. Methodology
  10. Objectives of Behavioral Calibration
  11. Three Strategies for Behavioral Calibration
  12. Explicit Risk Thresholding
  13. Verbalized Confidence
  14. Uniform Distribution.
  15. Beta Distribution.
  16. ...and 17 more sections

Figures (8)

  • Figure 1: Accuracy, hallucination, and abstention rates across the training progress (Step 10, 20, 30, 40, 50, and 60) of Explicit Risk Thresholding. Base model: Qwen3-4B-Instruct. Trained by GRPO on the DAPO-Math-17k dataset DBLP:journals/corr/abs-2503-14476. Evaluated on AIME 2024. Baseline is trained with the standard binary reward.
  • Figure 2: Sample output of Verbalized Confidence for individual claims from Qwen3-4B-Instruct-confidence-min.
  • Figure 3: Sample output of Critic Value as token-level confidence. The continuous confidence is visualized with a color gradient scale, ranging from red (0% confident) to green (100% confident). Evaluated on AIME-2024.
  • Figure 4: The calibration diagram for response-level stated confidence on BeyondAIME. The size of curves indicates density.
  • Figure 5: The calibration diagram for claim-level stated confidence on BeyondAIME. The size of curves indicates density. Note that in (a), the curves of Qwen3-Max and Gemini-2.5-Pro are overlapped.
  • ...and 3 more figures