Lie to Me: Knowledge Graphs for Robust Hallucination Self-Detection in LLMs

TL;DR

The paper addresses the persistent challenge of hallucinations in LLMs by introducing a knowledge-graph–based self-detection framework that converts model outputs into structured entity–relation graphs and uses these graphs to assess factuality. It integrates KG representations into both single-sample and multi-sample detection pipelines, evaluates on GPT-4o and Gemini-2.5-Flash across SimpleQA and WikiBio GPT-4o datasets, and releases a manually curated WikiBio GPT-4o dataset. Key contributions include a practical KG construction method from outputs, KG-enhanced variants of Self-Questioning, Self-Confidence, and SelfCheckGPT, and a thorough empirical analysis showing up to 16% relative accuracy gains and 20% F1-score improvements. The approach is low-cost, model-agnostic, and improves interpretability by enabling fact-level analysis of LLM outputs, advancing safer and more trustworthy language models.

Abstract

Hallucinations, the generation of apparently convincing yet false statements, remain a major barrier to the safe deployment of LLMs. Building on the strong performance of self-detection methods, we examine the use of structured knowledge representations, namely knowledge graphs, to improve hallucination self-detection. Specifically, we propose a simple yet powerful approach that enriches hallucination self-detection by (i) converting LLM responses into knowledge graphs of entities and relations, and (ii) using these graphs to estimate the likelihood that a response contains hallucinations. We evaluate the proposed approach using two widely used LLMs, GPT-4o and Gemini-2.5-Flash, across two hallucination detection datasets. To support more reliable future benchmarking, one of these datasets has been manually curated and enhanced and is released as a secondary outcome of this work. Compared to standard self-detection methods and SelfCheckGPT, a state-of-the-art approach, our method achieves up to 16% relative improvement in accuracy and 20% in F1-score. Our results show that LLMs can better analyse atomic facts when they are structured as knowledge graphs, even when initial outputs contain inaccuracies. This low-cost, model-agnostic approach paves the way toward safer and more trustworthy language models.

Figures (2)

• Figure 1: Single-sample hallucination self-detection flow for LLMs and the proposed knowledge graph augmented method.
• Figure 2: Multi sample hallucination self-detection flow for LLMs and the proposed knowledge graph augmented method.