AgentHallu: Benchmarking Automated Hallucination Attribution of LLM-based Agents

TL;DR

The paper tackles the problem of identifying where and why hallucinations originate in multi-step LLM-based agents and introduces AgentHallu, a comprehensive benchmark with 693 trajectories, 7 agent frameworks, 5 knowledge domains, and a fine-grained 5-category/14-subcategory taxonomy plus multi-level annotations. It formalizes two objectives—Hallucination Judgment and Hallucination Attribution—and provides an evaluation framework using step localization accuracy and G-EVAL explanations across 13 LLMs and two prompting methods. Empirical results reveal a large gap between judgment and attribution (best localization 41.1% and tool-use 11.6%), with attribution performance deteriorating as trajectory length grows and open-source models lagging behind proprietary ones; thinking-mode prompts offer some improvements. The work demonstrates substantial challenges in reliable, transparent agentic reasoning and offers a principled dataset and methodology to drive future research on robust, explainable LLM-based agents.

Abstract

As LLM-based agents operate over sequential multi-step reasoning, hallucinations arising at intermediate steps risk propagating along the trajectory, thus degrading overall reliability. Unlike hallucination detection in single-turn responses, diagnosing hallucinations in multi-step workflows requires identifying which step causes the initial divergence. To fill this gap, we propose a new research task, automated hallucination attribution of LLM-based agents, aiming to identify the step responsible for the hallucination and explain why. To support this task, we introduce AgentHallu, a comprehensive benchmark with: (1) 693 high-quality trajectories spanning 7 agent frameworks and 5 domains, (2) a hallucination taxonomy organized into 5 categories (Planning, Retrieval, Reasoning, Human-Interaction, and Tool-Use) and 14 sub-categories, and (3) multi-level annotations curated by humans, covering binary labels, hallucination-responsible steps, and causal explanations. We evaluate 13 leading models, and results show the task is challenging even for top-tier models (like GPT-5, Gemini-2.5-Pro). The best-performing model achieves only 41.1\% step localization accuracy, where tool-use hallucinations are the most challenging at just 11.6\%. We believe AgentHallu will catalyze future research into developing robust, transparent, and reliable agentic systems.

Figures (21)

• Figure 1: Illustration of hallucination attribution in LLM-based agents. Left: A misdefinition of regions X, Y, Z in Step 1 propagates to the tool call and leads to the incorrect final answer. Right: Beyond binary judgment, hallucination attribution aims to identify a hallucination-responsible step and a causal explanation.
• Figure 2: Overview of hallucination taxonomy in AgentHallu. The dataset includes 5 hallucination categories and 14 subcategories, where each trajectory step interleaves a thought step, an action step, and an observation step.
• Figure 3: Comparison of hallucination judgment and attribution performance across LLMs under varying trajectory steps $N_{\text{step}}$. Level 1 spans trajectories with $N_{\text{step}} \le 5$, Level 2 spans $6 \le N_{\text{step}} \le 10$, and Level 3 spans $N_{\text{step}} \ge 11$.
• Figure 4: Comparison of hallucination judgment F1 (%) on AgentHallu against existing hallucination detection datasets across multiple LLMs.
• Figure 5: The performance of judgment and attribution with and without thinking mode on Qwen3.