Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspective

Zhongxiang Sun, Qipeng Wang, Haoyu Wang, Xiao Zhang, Jun Xu

TL;DR

This work identifies reasoning hallucination as a safety risk in Large Reasoning Models and provides a mechanistic framework to study it. It introduces the Reasoning Score, a step-level metric derived from late-layer representation transformations via LogitLens, to distinguish deep reasoning from shallow pattern-matching. The authors formulate Reasoning Hallucination Detection (RHD) and propose GRPO-R, a step-level reward shaping approach, backed by theoretical generalization bounds and empirical gains on the ReTruthQA benchmark. Across math, science, and multi-hop domains, the approach improves detection accuracy and reduces hallucination rates, with practical implications for robust multi-step reasoning in open-source LRMs.

Abstract

Large Reasoning Models (LRMs) have shown impressive capabilities in multi-step reasoning tasks. However, alongside these successes, a more deceptive form of model error has emerged--Reasoning Hallucination--where logically coherent but factually incorrect reasoning traces lead to persuasive yet faulty conclusions. Unlike traditional hallucinations, these errors are embedded within structured reasoning, making them more difficult to detect and potentially more harmful. In this work, we investigate reasoning hallucinations from a mechanistic perspective. We propose the Reasoning Score, which quantifies the depth of reasoning by measuring the divergence between logits obtained from projecting late layers of LRMs to the vocabulary space, effectively distinguishing shallow pattern-matching from genuine deep reasoning. Using this score, we conduct an in-depth analysis on the ReTruthQA dataset and identify two key reasoning hallucination patterns: early-stage fluctuation in reasoning depth and incorrect backtracking to flawed prior steps. These insights motivate our Reasoning Hallucination Detection (RHD) framework, which achieves state-of-the-art performance across multiple domains. To mitigate reasoning hallucinations, we further introduce GRPO-R, an enhanced reinforcement learning algorithm that incorporates step-level deep reasoning rewards via potential-based shaping. Our theoretical analysis establishes stronger generalization guarantees, and experiments demonstrate improved reasoning quality and reduced hallucination rates.

Detection and Mitigation of Hallucination in Large Reasoning Models: A Mechanistic Perspective

TL;DR

This work identifies reasoning hallucination as a safety risk in Large Reasoning Models and provides a mechanistic framework to study it. It introduces the Reasoning Score, a step-level metric derived from late-layer representation transformations via LogitLens, to distinguish deep reasoning from shallow pattern-matching. The authors formulate Reasoning Hallucination Detection (RHD) and propose GRPO-R, a step-level reward shaping approach, backed by theoretical generalization bounds and empirical gains on the ReTruthQA benchmark. Across math, science, and multi-hop domains, the approach improves detection accuracy and reduces hallucination rates, with practical implications for robust multi-step reasoning in open-source LRMs.

Abstract

Large Reasoning Models (LRMs) have shown impressive capabilities in multi-step reasoning tasks. However, alongside these successes, a more deceptive form of model error has emerged--Reasoning Hallucination--where logically coherent but factually incorrect reasoning traces lead to persuasive yet faulty conclusions. Unlike traditional hallucinations, these errors are embedded within structured reasoning, making them more difficult to detect and potentially more harmful. In this work, we investigate reasoning hallucinations from a mechanistic perspective. We propose the Reasoning Score, which quantifies the depth of reasoning by measuring the divergence between logits obtained from projecting late layers of LRMs to the vocabulary space, effectively distinguishing shallow pattern-matching from genuine deep reasoning. Using this score, we conduct an in-depth analysis on the ReTruthQA dataset and identify two key reasoning hallucination patterns: early-stage fluctuation in reasoning depth and incorrect backtracking to flawed prior steps. These insights motivate our Reasoning Hallucination Detection (RHD) framework, which achieves state-of-the-art performance across multiple domains. To mitigate reasoning hallucinations, we further introduce GRPO-R, an enhanced reinforcement learning algorithm that incorporates step-level deep reasoning rewards via potential-based shaping. Our theoretical analysis establishes stronger generalization guarantees, and experiments demonstrate improved reasoning quality and reduced hallucination rates.
Paper Structure (41 sections, 1 theorem, 40 equations, 8 figures, 6 tables)

This paper contains 41 sections, 1 theorem, 40 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Empirical Study of Reasoning Hallucination
  4. Reasoning Score: Measuring Reasoning Depth in Large Reasoning Model
  5. Validating the Reasoning Score with GSM-NoOp.
  6. Results.
  7. Reasoning Hallucination Analysis Based on Reasoning Score
  8. Case Analysis on GSM-NoOps
  9. Reasoning Hallucination Pattern Analysis
  10. Results.
  11. Analyzing the Mechanisms Behind Reasoning Fluctuation
  12. Analysis.
  13. Methods
  14. Reasoning Hallucination Detection
  15. Mitigating Reasoning Hallucinations via Step-Level Reasoning Score Shaping
  16. ...and 26 more sections

Key Result

Theorem 1

Let the policy class $\Pi$ be such that for any $\pi \in \Pi$, the augmented return $R(\pi,\xi) = \sum_{t=1}^{T} \gamma^{t-1} \bar{r}_t(\xi)$ is uniformly bounded in $[0, \bar{R}_{\max}]$ for any trajectory $\xi$ sampled from the environment. Each trajectory $\xi = (s_1, a_1, \bar{r}_1, \dots, s_T, where $J_{\text{test}}(\pi)=\mathbb{E}_{\xi}[R(\pi,\xi)]$ is the expected test return and $J_{\text

Figures (8)

  • Figure 1: The illustration of the calculation processes for the Reasoning Score (Eq. \ref{['eq:r_score']}), CV Score (Eq. \ref{['eq:cv_score']}), and Attention Score (Eq. \ref{['eq:attn_score']}).
  • Figure 2: Case study from GSM-NoOp dataset mirzadeh2024gsm on R1-7B. We sample both a hallucinated reasoning trace (left) and a truthful reasoning trace (right) for the same question as a preliminary analysis of reasoning hallucinations. Reasoning scores are scaled by $1\mathrm{e}{5}$.
  • Figure 3: (a) Reasoning Score validation on GSM-NoOp. (b) Evaluation of Pattern #1 (early fluctuations), and (c) Pattern #2 (misguidedly attention) on ReTruthQA. Asterisks indicate statistical significance based on a t-test: * for $p$-value < 0.05, and *** for $p$-value < 0.001.
  • Figure 4: Analysis of Pattern #1: (a) Consistency Analysis (Q1); (b) Accuracy Comparison in Rising-2 triples (Q2); (c) Reasoning score vs. perplexity and (d) Perplexity of Rising-2 vs. Stable (Q3).
  • Figure 5: Interface Display of the Data Annotation Platform.
  • ...and 3 more figures

Theorems & Definitions (2)

  • Theorem 1: Generalization Gap with Augmented Rewards
  • proof : Proof of Theorem \ref{['thm:LRM:gb']}