Hallucination Detection: A Probabilistic Framework Using Embeddings Distance Analysis

TL;DR

Hallucinations in large language models undermine reliability in production. This work introduces a probabilistic framework that detects hallucinations by analyzing distances between semantic embeddings using Minkowski norms to reveal structural differences between genuine and hallucinated content. A two-model, controlled dataset (Llama2 vs Llama3) demonstrates statistically significant, scale-free separation in embedding-distance distributions and yields a detector with about 66% accuracy, serving as a promising PoC. The approach provides a principled, transferable method for detection and suggests directions for refinement with larger models and real-world data.

Abstract

Hallucinations are one of the major issues affecting LLMs, hindering their wide adoption in production systems. While current research solutions for detecting hallucinations are mainly based on heuristics, in this paper we introduce a mathematically sound methodology to reason about hallucination, and leverage it to build a tool to detect hallucinations. To the best of our knowledge, we are the first to show that hallucinated content has structural differences with respect to correct content. To prove this result, we resort to the Minkowski distances in the embedding space. Our findings demonstrate statistically significant differences in the embedding distance distributions, that are also scale free -- they qualitatively hold regardless of the distance norm used and the number of keywords, questions, or responses. We leverage these structural differences to develop a tool to detect hallucinated responses, achieving an accuracy of 66\% for a specific configuration of system parameters -- comparable with the best results in the field. In conclusion, the suggested methodology is promising and novel, possibly paving the way for further research in the domain, also along the directions highlighted in our future work.

Figures (16)

• Figure 1: Timeline showing---in green---the periods covered by model training for Llama2 and Llama3. The questions used for experiments rely on facts that happened between September 2022 and September 2023, a training period covered by Llama3, but not by Llama2.
• Figure 2: Overview of the methodology implemented to extract information from the training responses.
• Figure 3: Overview of the methodology implemented to classify test responses.
• Figure 4: Training distributions of Minkowski distances with $r=16$. Statistical significance: ***: $p < 0.01$, **: $p < 0.05$, *: $p < 0.1$.
• Figure 5: KL Divergence and median Difference vs $n$ at different $r$ for $p=2.0$ using training data.