Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models

Junyi Li, Hwee Tou Ng

TL;DR

This work identifies that RL fine-tuning for large reasoning models exacerbates hallucinations due to high-variance policy gradients, entropy-driven exploration, and spurious local optima under outcome-based rewards. To counter this, it introduces Factuality-aware Step-wise Policy Optimization (FSPO), which injects step-wise factual verification into the RL loop to adjust token-level advantages toward factually grounded reasoning. FSPO combines an entailment-based step-wise factuality reward with the final answer reward and reweights token advantages per step, enabling denser and more informative feedback than final-outcome signals. Empirical results across mathematical reasoning and hallucination benchmarks on Qwen2.5 and Llama models show that FSPO reduces hallucinations while improving reasoning accuracy, outperforming several open-source baselines and demonstrating robustness across tasks. The approach advances the reliability of RL-tuned reasoning systems by explicitly prioritizing verifiable reasoning steps over merely correct final answers.

Abstract

Large language models (LLMs) have significantly advanced in reasoning tasks through reinforcement learning (RL) optimization, achieving impressive capabilities across various challenging benchmarks. However, our empirical analysis reveals a critical drawback: reasoning-oriented RL fine-tuning significantly increases the prevalence of hallucinations. We theoretically analyze the RL training dynamics, identifying high-variance gradient, entropy-induced randomness, and susceptibility to spurious local optima as key factors leading to hallucinations. To address this drawback, we propose Factuality-aware Step-wise Policy Optimization (FSPO), an innovative RL fine-tuning algorithm incorporating explicit factuality verification at each reasoning step. FSPO leverages automated verification against given evidence to dynamically adjust token-level advantage values, incentivizing factual correctness throughout the reasoning process. Experiments across mathematical reasoning and hallucination benchmarks using Qwen2.5 and Llama models demonstrate that FSPO effectively reduces hallucinations while enhancing reasoning accuracy, substantially improving both reliability and performance.

Reasoning Models Hallucinate More: Factuality-Aware Reinforcement Learning for Large Reasoning Models

TL;DR

This work identifies that RL fine-tuning for large reasoning models exacerbates hallucinations due to high-variance policy gradients, entropy-driven exploration, and spurious local optima under outcome-based rewards. To counter this, it introduces Factuality-aware Step-wise Policy Optimization (FSPO), which injects step-wise factual verification into the RL loop to adjust token-level advantages toward factually grounded reasoning. FSPO combines an entailment-based step-wise factuality reward with the final answer reward and reweights token advantages per step, enabling denser and more informative feedback than final-outcome signals. Empirical results across mathematical reasoning and hallucination benchmarks on Qwen2.5 and Llama models show that FSPO reduces hallucinations while improving reasoning accuracy, outperforming several open-source baselines and demonstrating robustness across tasks. The approach advances the reliability of RL-tuned reasoning systems by explicitly prioritizing verifiable reasoning steps over merely correct final answers.

Abstract

Large language models (LLMs) have significantly advanced in reasoning tasks through reinforcement learning (RL) optimization, achieving impressive capabilities across various challenging benchmarks. However, our empirical analysis reveals a critical drawback: reasoning-oriented RL fine-tuning significantly increases the prevalence of hallucinations. We theoretically analyze the RL training dynamics, identifying high-variance gradient, entropy-induced randomness, and susceptibility to spurious local optima as key factors leading to hallucinations. To address this drawback, we propose Factuality-aware Step-wise Policy Optimization (FSPO), an innovative RL fine-tuning algorithm incorporating explicit factuality verification at each reasoning step. FSPO leverages automated verification against given evidence to dynamically adjust token-level advantage values, incentivizing factual correctness throughout the reasoning process. Experiments across mathematical reasoning and hallucination benchmarks using Qwen2.5 and Llama models demonstrate that FSPO effectively reduces hallucinations while enhancing reasoning accuracy, substantially improving both reliability and performance.

Paper Structure

This paper contains 20 sections, 3 theorems, 12 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminary Experiments
  3. Background
  4. Theoretical Analysis
  5. Approach
  6. Reward Function Design
  7. Factuality-Aware Policy Optimization
  8. Experiments
  9. Experimental Setup
  10. Main Results
  11. Further Analysis
  12. Related Work
  13. Conclusion
  14. Acknowledgments
  15. Limitations
  16. ...and 5 more sections

Key Result

Theorem 4.1

Under a binary reward function, the variance of the policy gradient estimator is directly proportional to the probability of generating correct answers. Let $p$ denote the probability of producing a correct answer, so the variance of the policy gradient estimator is on the order of $p(1-p)\|\nabla_{

Figures (7)

  • Figure 1: Hallucination rate (%) for six groups of twelve LLMs across three benchmarks.
  • Figure 2: Distribution of hallucination causes over 200 samples from HaluEval-QA.
  • Figure 3: Overall framework of our proposed FSPO approach.
  • Figure 4: Pass@1 accuracy over training steps for ablation analysis.
  • Figure 5: Pass@1 accuracy over training steps with different RL algorithms.
  • ...and 2 more figures

Theorems & Definitions (6)

  • Theorem 4.1: Variance of Policy Gradient in Binary Reward RL
  • proof
  • Theorem 4.2: Exploration-Exploitation Trade-off and Hallucination Risk
  • proof
  • Theorem 4.3: Local Optima and Incorrect Convergence
  • proof