HALT: Hallucination Assessment via Latent Testing

TL;DR

HALT introduces a latent-reading approach to hallucination detection that operates on intermediate question-token representations to estimate risk without adding generation latency. A lightweight detector (MLP or Transformer) maps hidden states to a probability of correct answers, enabling a router to dispatch high-risk queries to more reliable pipelines while low-risk queries proceed with standard generation. Across four open QA benchmarks and multiple model families, HALT achieves strong AUROC and AURAC, demonstrates robustness under distribution shift, and reveals that intermediate layers retain informative uncertainty signals. The method offers a principled foundation for fast, reliable agentic AI by combining real-time risk readouts with selective deployment and targeted verification, potentially informing future RL and search-based AI systems.

Abstract

Hallucination in large language models (LLMs) can be understood as a failure of faithful readout: although internal representations may encode uncertainty about a query, decoding pressures still yield a fluent answer. We propose lightweight residual probes that read hallucination risk directly from intermediate hidden states of question tokens, motivated by the hypothesis that these layers retain epistemic signals that are attenuated in the final decoding stage. The probe is a small auxiliary network whose computation is orders of magnitude cheaper than token generation and can be evaluated fully in parallel with inference, enabling near-instantaneous hallucination risk estimation with effectively zero added latency in low-risk cases. We deploy the probe as an agentic critic for fast selective generation and routing, allowing LLMs to immediately answer confident queries while delegating uncertain ones to stronger verification pipelines. Across four QA benchmarks and multiple LLM families, the method achieves strong AUROC and AURAC, generalizes under dataset shift, and reveals interpretable structure in intermediate representations, positioning fast internal uncertainty readout as a principled foundation for reliable agentic AI.

Figures (8)

• Figure 1: Comparison of hallucination handling strategies. Traditional detection pipelines wait for generation to complete before checking for hallucinations, doubling latency in the fallback case. Our method evaluates hallucination risk in parallel with generation, enabling zero-latency responses for confident queries while routing uncertain ones to stronger models.
• Figure 2: Out-of-distribution generalization. Heatmap of AUROC when training the detector on one dataset (columns) and evaluating it on another (rows), including training on the union of all datasets (All).
• Figure 3: Rejection-Accuracy Curve (RAC) for Qwen-2.5-7B on TriviaQA using only question token representations. As we reject samples with low confidence, accuracy increases from 85.7% at full coverage to 97.7% at 40% coverage. AURAC = 0.8753 indicates strong alignment between confidence and correctness.
• Figure 4: AUROC and AURAC across layers 0--28 for Qwen2.5-7B on Webquestions. Both metrics peak around layer 20, suggesting that intermediate-to-late layers encode the most informative signals for uncertainty detection.
• Figure 5: Effect of model scale (7B, 14B, 32B) on AUROC and AURAC. While AUROC remains largely stable across scales, AURAC increases consistently, indicating improved ranking quality under selective deployment for larger models.