Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HALT: Hallucination Assessment via Log-probs as Time series

Ahmad Shapiro, Karan Taneja, Ashok Goel

TL;DR

HALT addresses hallucinations in LLMs by leveraging only token-level log-probabilities as a time series. It treats top-$k$ log-probabilities ($k=20$) as a proximal distribution and trains a compact bidirectional GRU with engineered features to detect calibration biases that accompany incorrect outputs. The HUB benchmark consolidates ten capabilities, enabling macro-F1 evaluation and cross-model analysis; results show HALT outperforms surface-text or simple-statistics baselines and demonstrates model-specific calibration dynamics with partial cross-model transferability. This approach offers a lightweight, privacy-preserving detector suitable for API-only deployments and provides a foundation for real-time monitoring and calibration-aware decoding in diverse domains.

Abstract

Hallucinations remain a major obstacle for large language models (LLMs), especially in safety-critical domains. We present HALT (Hallucination Assessment via Log-probs as Time series), a lightweight hallucination detector that leverages only the top-20 token log-probabilities from LLM generations as a time series. HALT uses a gated recurrent unit model combined with entropy-based features to learn model calibration bias, providing an extremely efficient alternative to large encoders. Unlike white-box approaches, HALT does not require access to hidden states or attention maps, relying only on output log-probabilities. Unlike black-box approaches, it operates on log-probs rather than surface-form text, which enables stronger domain generalization and compatibility with proprietary LLMs without requiring access to internal weights. To benchmark performance, we introduce HUB (Hallucination detection Unified Benchmark), which consolidates prior datasets into ten capabilities covering both reasoning tasks (Algorithmic, Commonsense, Mathematical, Symbolic, Code Generation) and general purpose skills (Chat, Data-to-Text, Question Answering, Summarization, World Knowledge). While being 30x smaller, HALT outperforms Lettuce, a fine-tuned modernBERT-base encoder, achieving a 60x speedup gain on HUB. HALT and HUB together establish an effective framework for hallucination detection across diverse LLM capabilities.

HALT: Hallucination Assessment via Log-probs as Time series

TL;DR

HALT addresses hallucinations in LLMs by leveraging only token-level log-probabilities as a time series. It treats top- log-probabilities () as a proximal distribution and trains a compact bidirectional GRU with engineered features to detect calibration biases that accompany incorrect outputs. The HUB benchmark consolidates ten capabilities, enabling macro-F1 evaluation and cross-model analysis; results show HALT outperforms surface-text or simple-statistics baselines and demonstrates model-specific calibration dynamics with partial cross-model transferability. This approach offers a lightweight, privacy-preserving detector suitable for API-only deployments and provides a foundation for real-time monitoring and calibration-aware decoding in diverse domains.

Abstract

Hallucinations remain a major obstacle for large language models (LLMs), especially in safety-critical domains. We present HALT (Hallucination Assessment via Log-probs as Time series), a lightweight hallucination detector that leverages only the top-20 token log-probabilities from LLM generations as a time series. HALT uses a gated recurrent unit model combined with entropy-based features to learn model calibration bias, providing an extremely efficient alternative to large encoders. Unlike white-box approaches, HALT does not require access to hidden states or attention maps, relying only on output log-probabilities. Unlike black-box approaches, it operates on log-probs rather than surface-form text, which enables stronger domain generalization and compatibility with proprietary LLMs without requiring access to internal weights. To benchmark performance, we introduce HUB (Hallucination detection Unified Benchmark), which consolidates prior datasets into ten capabilities covering both reasoning tasks (Algorithmic, Commonsense, Mathematical, Symbolic, Code Generation) and general purpose skills (Chat, Data-to-Text, Question Answering, Summarization, World Knowledge). While being 30x smaller, HALT outperforms Lettuce, a fine-tuned modernBERT-base encoder, achieving a 60x speedup gain on HUB. HALT and HUB together establish an effective framework for hallucination detection across diverse LLM capabilities.
Paper Structure (54 sections, 23 equations, 2 figures, 10 tables)

This paper contains 54 sections, 23 equations, 2 figures, 10 tables.

Table of Contents

  1. Introduction
  2. HUB: Hallucination detection Unified Benchmark
  3. Scope
  4. Splits and Generalization Protocol
  5. Analysis
  6. Methodology
  7. Motivation
  8. Hypothesis 1 (Model-Specific Bias).
  9. Hypothesis 2 (Bias Embedding and Learnability).
  10. Hypothesis 3 (Non-Transferability Across Models).
  11. Hypothesis 4 (Task Generalization).
  12. Approach
  13. Feature Extraction.
  14. From top-$20$ log-probs to a proximal distribution.
  15. Selected features.
  16. ...and 39 more sections

Figures (2)

  • Figure 1: HALT Architecture. The system extracts log-prob sequences from the LLM. The matrix visualizes the input $\boldsymbol{\ell}_{1:T}$ with tokens $t_1 \dots t_T$ and top-20 log-probs $\ell_1 \dots \ell_{20}$. These are concatenated with engineered features and processed by a lightweight GRU.
  • Figure 2: Example of a Logical Hallucination. The model correctly identifies the premises but fails the internal arithmetic reasoning step (the numbers shown actually sum to 13, and the true total is 14). This sample originates from the Algorithmic Reasoning cluster in HUB, specifically the ObjectCounting shard of the CriticBench dataset.