Internal Representations as Indicators of Hallucinations in Agent Tool Selection
Kait Healy, Bharathi Srinivasan, Visakh Madathil, Jing Wu
TL;DR
The paper addresses tool-calling hallucinations in LLM-based agents by introducing a real-time detector that uses the model’s last-layer transformer representations during the generation pass. It formalizes the problem as learning a binary classifier $h_\theta: \mathbb{R}^d \rightarrow \{0,1\}$ over a feature $\mathbf{z}$ constructed from the final-layer states around the predicted tool call, with $p_i=h_\theta(\mathbf{z}_i)$ and a decision threshold $\tau$. An unsupervised data-collection pipeline masks ground-truth tool calls and labels predictions by agreement, enabling single-pass, low-overhead training of a lightweight MLP; during inference, the detector gates execution to prevent or repair hallucinations. Across GPT-OSS-20B, Llama-3.1-8B, and Qwen7B on multi-domain tool-calling data, the method achieves up to $86\%$ accuracy and demonstrates strong discrimination between correct and hallucinated calls, with ablations showing mean-pooling of final-layer representations as a strong, simple baseline. The work advances practical deployment by enabling real-time risk scores within production agents, though recall varies by model and labeling alignment remains a challenge for cross-model generalization.
Abstract
Large Language Models (LLMs) have shown remarkable capabilities in tool calling and tool usage, but suffer from hallucinations where they choose incorrect tools, provide malformed parameters and exhibit 'tool bypass' behavior by performing simulations and generating outputs instead of invoking specialized tools or external systems. This undermines the reliability of LLM based agents in production systems as it leads to inconsistent results, and bypasses security and audit controls. Such hallucinations in agent tool selection require early detection and error handling. Unlike existing hallucination detection methods that require multiple forward passes or external validation, we present a computationally efficient framework that detects tool-calling hallucinations in real-time by leveraging LLMs' internal representations during the same forward pass used for generation. We evaluate this approach on reasoning tasks across multiple domains, demonstrating strong detection performance (up to 86.4\% accuracy) while maintaining real-time inference capabilities with minimal computational overhead, particularly excelling at detecting parameter-level hallucinations and inappropriate tool selections, critical for reliable agent deployment.