Joint Evaluation of Answer and Reasoning Consistency for Hallucination Detection in Large Reasoning Models

TL;DR

RACE introduces a black-box framework for detecting hallucinations in Large Reasoning Models by jointly evaluating the reasoning traces and final answers. Grounded in an information-theoretic formulation of joint uncertainty, it distills reasoning traces into concise chains of thought and computes four signals—Reasoning Consistency $S_{CC}$, Answer Uncertainty $S_{AA}$, Reasoning–Answer Alignment $S_{CA}$, and Reasoning Internal Coherence $S_{Coh}$—which are aggregated into a final score $S_{RACE}$. Across multiple datasets and LLM families, RACE outperforms answer-only baselines and remains robust under different sampling regimes, with ablations confirming the complementary value of each component. The approach supports broad applicability in black-box settings and offers a path toward improved reliability of LRMs, potentially extending to retrieval-augmented generation and inference-time mitigation strategies. The authors also release an open-source CoT extraction module to distill relevant reasoning steps and enhance the practicality of reasoning-aware evaluation.

Abstract

Large Reasoning Models (LRMs) extend large language models with explicit, multi-step reasoning traces to enhance transparency and performance on complex tasks. However, these reasoning traces can be redundant or logically inconsistent, becoming a new and hard-to-detect source of hallucination. Existing hallucination detection methods focus primarily on answer-level uncertainty and often fail to detect hallucinations or logical inconsistencies arising from the model's reasoning trace. This oversight is particularly problematic for LRMs, where the explicit thinking trace is not only an important support to the model's decision-making process but also a key source of potential hallucination. To this end, we propose RACE (Reasoning and Answer Consistency Evaluation), a novel framework specifically tailored for hallucination detection in LRMs. RACE operates by extracting essential reasoning steps and computing four diagnostic signals: inter-sample consistency of reasoning traces, entropy-based answer uncertainty, semantic alignment between reasoning and answers, and internal coherence of reasoning. The joint utilization of these signals makes RACE a more robust detector of hallucinations in LRMs. Experiments across datasets and different LLMs demonstrate that RACE outperforms existing hallucination detection baselines, offering a robust and generalizable solution for evaluating LRMs. The source code is available at https://github.com/bebr2/RACE

Figures (2)

• Figure 1: This figure illustrates the importance of incorporating reasoning paths in hallucination detection for LRMs. The example is from HotpotQA. The model is DeepSeek-R1-Distill-Qwen7B. While three of the four final answers mention “New York City,” the underlying reasoning traces reveal divergent and often inaccurate chains of logic. Red highlights indicate hallucinated or unsupported claims that are absent from the final answers. Thus, evaluating only the answer-level output would misleadingly suggest that the model is consistent with the answer, underscoring the need for reasoning-aware hallucination detection.
• Figure 2: AUROC performance of RACE and SINdex across different numbers of generations on NQ-Open.