Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Energy of Falsehood: Detecting Hallucinations via Diffusion Model Likelihoods

Arpit Singh Gautam, Kailash Talreja, Saurabh Jha

TL;DR

DiffuTruth tackles LLM hallucinations by treating factual truth as a stable attractor on a learned text-diffusion manifold and by probing semantic stability under controlled noise with a Generative Stress Test. It defines Semantic Energy via an NLI critic to quantify semantic drift between original and reconstructed claims, and blends this generative signal with a discriminative baseline through Hybrid Calibration. Empirical results on FEVER show a 0.725 AUROC with unsupervised setup, and superior zero-shot generalization on HOVER, indicating robust performance under distribution shifts. The framework highlights the value of generative stability in fact verification, albeit with higher computational costs and reliance on true-statements for training, suggesting directions for scaling, real-time deployment, and multi-modal extensions.

Abstract

Large Language Models (LLMs) frequently hallucinate plausible but incorrect assertions, a vulnerability often missed by uncertainty metrics when models are confidently wrong. We propose DiffuTruth, an unsupervised framework that reconceptualizes fact verification via non equilibrium thermodynamics, positing that factual truths act as stable attractors on a generative manifold while hallucinations are unstable. We introduce the Generative Stress Test, claims are corrupted with noise and reconstructed using a discrete text diffusion model. We define Semantic Energy, a metric measuring the semantic divergence between the original claim and its reconstruction using an NLI critic. Unlike vector space errors, Semantic Energy isolates deep factual contradictions. We further propose a Hybrid Calibration fusing this stability signal with discriminative confidence. Extensive experiments on FEVER demonstrate DiffuTruth achieves a state of the art unsupervised AUROC of 0.725, outperforming baselines by 1.5 percent through the correction of overconfident predictions. Furthermore, we show superior zero shot generalization on the multi hop HOVER dataset, outperforming baselines by over 4 percent, confirming the robustness of thermodynamic truth properties to distribution shifts.

The Energy of Falsehood: Detecting Hallucinations via Diffusion Model Likelihoods

TL;DR

DiffuTruth tackles LLM hallucinations by treating factual truth as a stable attractor on a learned text-diffusion manifold and by probing semantic stability under controlled noise with a Generative Stress Test. It defines Semantic Energy via an NLI critic to quantify semantic drift between original and reconstructed claims, and blends this generative signal with a discriminative baseline through Hybrid Calibration. Empirical results on FEVER show a 0.725 AUROC with unsupervised setup, and superior zero-shot generalization on HOVER, indicating robust performance under distribution shifts. The framework highlights the value of generative stability in fact verification, albeit with higher computational costs and reliance on true-statements for training, suggesting directions for scaling, real-time deployment, and multi-modal extensions.

Abstract

Large Language Models (LLMs) frequently hallucinate plausible but incorrect assertions, a vulnerability often missed by uncertainty metrics when models are confidently wrong. We propose DiffuTruth, an unsupervised framework that reconceptualizes fact verification via non equilibrium thermodynamics, positing that factual truths act as stable attractors on a generative manifold while hallucinations are unstable. We introduce the Generative Stress Test, claims are corrupted with noise and reconstructed using a discrete text diffusion model. We define Semantic Energy, a metric measuring the semantic divergence between the original claim and its reconstruction using an NLI critic. Unlike vector space errors, Semantic Energy isolates deep factual contradictions. We further propose a Hybrid Calibration fusing this stability signal with discriminative confidence. Extensive experiments on FEVER demonstrate DiffuTruth achieves a state of the art unsupervised AUROC of 0.725, outperforming baselines by 1.5 percent through the correction of overconfident predictions. Furthermore, we show superior zero shot generalization on the multi hop HOVER dataset, outperforming baselines by over 4 percent, confirming the robustness of thermodynamic truth properties to distribution shifts.
Paper Structure (32 sections, 3 equations, 3 figures, 7 tables)

This paper contains 32 sections, 3 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Theoretical Framework: Truth as an Attractor
  5. The Generative Stress Test
  6. Hybrid Calibration
  7. Experimental Setup
  8. Results
  9. Main Results on FEVER
  10. Cross-Dataset Generalization (HOVER)
  11. Discussion
  12. Generative Calibration
  13. Robustness and the Topology of Truth
  14. Limitations and Future Directions
  15. Conclusion and Future Work
  16. ...and 17 more sections

Figures (3)

  • Figure 1: DiffuTruth architecture workflow.
  • Figure 2: Performance across datasets.
  • Figure 3: OOD generalization performance.