Learned Hallucination Detection in Black-Box LLMs using Token-level Entropy Production Rate

TL;DR

This paper addresses hallucinations in black-box LLMs by proposing a one-shot uncertainty framework that relies only on top-$K$ log-probabilities exposed by APIs. It develops an unsupervised Entropy Production Rate ($EPR$) and a supervised Weighted Entropy Production Rate ($WEPR$) to detect hallucinations at the token level in QA and RAG contexts. Across multiple models and datasets, $WEPR$ consistently outperforms existing black-box detectors and robustly handles small $K$ values (as low as $K\leq 10$), making it practical for API-constrained deployments. The approach provides both sentence-level and token-level uncertainty signals and is released as open-source for adoption in QA and RAG systems to improve trust in LLM outputs.

Abstract

Hallucinations in Large Language Model (LLM) outputs for Question Answering (QA) tasks can critically undermine their real-world reliability. This paper introduces a methodology for robust, one-shot hallucination detection, specifically designed for scenarios with limited data access, such as interacting with black-box LLM APIs that typically expose only a few top candidate log-probabilities per token. Our approach derives uncertainty indicators directly from these readily available log-probabilities generated during non-greedy decoding. We first derive an Entropy Production Rate (EPR) that offers baseline performance, later augmented with supervised learning. Our learned model leverages the entropic contributions of the accessible top-ranked tokens within a single generated sequence, without multiple re-runs per query. Evaluated across diverse QA datasets and multiple LLMs, this estimator significantly improves token-level hallucination detection over state-of-the-art methods. Crucially, high performance is demonstrated using only the typically small set of available log-probabilities (e.g., top-10 per token), confirming its practical efficiency and suitability for API-constrained deployments. This work provides a lightweight technique to enhance the trustworthiness of LLM responses, at the token level, after a single generation pass, for QA and Retrieval-Augmented Generation (RAG) systems. Our experiments confirmed the performance of our method against existing approaches on public dataset as well as for a financial framework analyzing annual company reports.

Figures (5)

• Figure 1: Illustration of token-level uncertainty detection in a financial RAG pipeline.
• Figure 2: Analysis of probability mass concentration on $180$k tokens generated by Mistral-Small-3.1-24B ($T_{\text{samp}}=1.0$). (a) Distribution of log-probabilities for the token at rank $k$, showing a rapid decay in likelihood. (b) Distribution of the missing probability mass $Q_K$ (log scale) when truncating at cutoff $K$. The median $Q_K$ drops below $10^{-4}$ by $K=10$, validating the Top-$K$ approximation.
• Figure 3: Distributions of EPR scores on Falcon-10B responses. The separation between the Hallucinated (red) and Non-hallucinated (green) distributions illustrates the discriminative power of the metric.
• Figure 4: Mean ROC-AUC with respect to $K$ for Falcon-3 on TriviaQA.
• Figure 5: Precision-Recall curves of 3 methods for Falcon-3 on TriviaQA.