The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucination

TL;DR

This work reveals a fundamental paradox: amplifying reasoning in LLM-based agents tends to increase tool hallucination, even when training targets are not tool-specific. Through SimpleToolHalluBench, the authors establish a causal link between reasoning reinforcement and tool hallucination, show that the effect generalizes across models and training methods, and identify late-layer residual streams as a critical locus for misleading divergences. Mechanistic analysis demonstrates that Reasoning RL destabilizes tool-related representations, while mitigation strategies incur a reliability-capability trade-off, with prompt engineering offering limited relief and DPO reducing hallucinations at the cost of tool-use performance. The findings argue for new training objectives that jointly optimize capability and reliability, including abstention and calibrated confidence, to build trustworthy, tool-aware LLM agents.

Abstract

Enhancing the reasoning capabilities of Large Language Models (LLMs) is a key strategy for building Agents that "think then act." However, recent observations, like OpenAI's o3, suggest a paradox: stronger reasoning often coincides with increased hallucination, yet no prior work has systematically examined whether reasoning enhancement itself causes tool hallucination. To address this gap, we pose the central question: Does strengthening reasoning increase tool hallucination? To answer this, we introduce SimpleToolHalluBench, a diagnostic benchmark measuring tool hallucination in two failure modes: (i) no tool available, and (ii) only distractor tools available. Through controlled experiments, we establish three key findings. First, we demonstrate a causal relationship: progressively enhancing reasoning through RL increases tool hallucination proportionally with task performance gains. Second, this effect transcends overfitting - training on non-tool tasks (e.g., mathematics) still amplifies subsequent tool hallucination. Third, the effect is method-agnostic, appearing when reasoning is instilled via supervised fine-tuning and when it is merely elicited at inference by switching from direct answers to step-by-step thinking. We also evaluate mitigation strategies including Prompt Engineering and Direct Preference Optimization (DPO), revealing a fundamental reliability-capability trade-off: reducing hallucination consistently degrades utility. Mechanistically, Reasoning RL disproportionately collapses tool-reliability-related representations, and hallucinations surface as amplified divergences concentrated in late-layer residual streams. These findings reveal that current reasoning enhancement methods inherently amplify tool hallucination, highlighting the need for new training objectives that jointly optimize for capability and reliability.

Figures (5)

• Figure 1: Overview of our key findings. Left: Reinforcement learning for reasoning enhancement increases tool hallucination rates alongside task performance gains. Middle: Mechanistic analysis reveals that reasoning RL destabilizes tool-reliability-related representations in the model's internal layers. Right: Mitigation strategies expose a fundamental trade-off—reducing hallucination consistently degrades utility, with no free lunch for achieving both reliability and capability.
• Figure 2: An overview of model performance during the training of ReCall chen2025learning. (a) Hallucination rate of different tasks during training. Lower is better. (b) Task-specific reward during training. Higher is better.
• Figure 3: Overview of model performance during GRPO training on GSM8K cobbe2021gsm8k. (a) Hallucination rate for different tasks throughout training (lower is better). (b) Task-specific reward during training (higher is better).
• Figure 4: Layer-wise representation stability after Reasoning RL. While in-distribution representations remain stable (green), tool representations collapse dramatically (blue).
• Figure 5: Component-wise discrimination scores across layers. The heatmap shows how distinguishable correct and hallucinated responses are within different model components. Residual stream components (resid_mid and resid_post) exhibit substantially higher discrimination scores in late layers (> 0.14), while attention and MLP outputs show consistently lower scores (< 0.08).