Lowest Span Confidence: A Zero-Shot Metric for Efficient and Black-Box Hallucination Detection in LLMs

TL;DR

This work tackles the challenge of hallucinations in large language models by introducing Lowest Span Confidence (LSC), a zero-shot, span-based detector that requires only a single forward pass and token probabilities. By sliding a window over token confidences and taking the minimum span-mean probability, LSC captures localized uncertainty that correlates with factual errors, avoiding both global dilution and token-level noise. Across four QA benchmarks and multiple model families, LSC consistently outperforms zero-shot baselines and remains practical for API-based deployment due to its efficiency. The results demonstrate that span-level uncertainty is a robust signal for detecting hallucinations, with strong implications for trustworthy and scalable LLM deployment in high-stakes settings.

Abstract

Hallucinations in Large Language Models (LLMs), i.e., the tendency to generate plausible but non-factual content, pose a significant challenge for their reliable deployment in high-stakes environments. However, existing hallucination detection methods generally operate under unrealistic assumptions, i.e., either requiring expensive intensive sampling strategies for consistency checks or white-box LLM states, which are unavailable or inefficient in common API-based scenarios. To this end, we propose a novel efficient zero-shot metric called Lowest Span Confidence (LSC) for hallucination detection under minimal resource assumptions, only requiring a single forward with output probabilities. Concretely, LSC evaluates the joint likelihood of semantically coherent spans via a sliding window mechanism. By identifying regions of lowest marginal confidence across variable-length n-grams, LSC could well capture local uncertainty patterns strongly correlated with factual inconsistency. Importantly, LSC can mitigate the dilution effect of perplexity and the noise sensitivity of minimum token probability, offering a more robust estimate of factual uncertainty. Extensive experiments across multiple state-of-the-art (SOTA) LLMs and diverse benchmarks show that LSC consistently outperforms existing zero-shot baselines, delivering strong detection performance even under resource-constrained conditions.

Figures (6)

• Figure 1: Comparison between LSC and Sampling-based Methods.
• Figure 2: Scalability analysis across model sizes.
• Figure 3: Ablation results of sliding window size $w$. The curves illustrate the impact of varying window size (from 1 to 8) on LSC detection performance across four datasets and four LLMs.
• Figure 4: Macroscopic performance evaluation via ROC curves across four representative LLMs. Comparison of LSC against the SOTA baseline EigenScore and window size variations ($k=1, 8$). Across different model families (LLaMA, Qwen) and scales (3B to 13B), LSC (solid red curve) consistently encloses the baselines. This demonstrates that LSC achieves the highest AUROC by maintaining a superior True Positive Rate while effectively suppressing False Positives.
• Figure 5: Qualitative comparison of specific cases.Left (Hallucination Sample): Selected to demonstrate how LSC correctly identifies errors where baselines suffer from false negatives. Right (Non-hallucination Sample): Selected to show LSC's robustness in verifying factual content where consistency-based baselines trigger false positives due to benign phrasing variations.