Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answering

TL;DR

This work introduces OKGQA, an open-ended KGQA benchmark to evaluate how integrating Knowledge Graphs can reduce hallucinations in Large Language Models during open-ended question answering. It formalizes a KG-augmented retrieval-generation framework with a two-component design (G-retrieval and G-Generator) and a formalization that combines retrieved knowledge Z with the LLM’s output via p(a|q) ≈ pφ(a|q,Z*) pθ(Z*|q,G). Through OKGQA and its perturbation variant OKGQA-P, the authors demonstrate that KG-informed methods, particularly subgraph-based retrieval, consistently reduce factual errors and enhance reasoning, even when the KG is noisy. The results underscore that relying solely on internal reasoning (CoT, Self-Consistency) is insufficient to curb hallucinations and that external structured knowledge provides robust gains across diverse query types and model scales. The work offers practical guidance for KG design and integration to improve trustworthiness in LLMs and points to future work on domain-specific and dynamic KG sources.

Abstract

Recent works integrating Knowledge Graphs (KGs) have shown promising improvements in enhancing the reasoning capabilities of Large Language Models (LLMs). However, existing benchmarks primarily focus on closed-ended tasks, leaving a gap in evaluating performance on more complex, real-world scenarios. This limitation also hinders a thorough assessment of KGs' potential to reduce hallucinations in LLMs. To address this, we introduce OKGQA, a new benchmark specifically designed to evaluate LLMs augmented with KGs in open-ended, real-world question answering settings. OKGQA reflects practical complexities through diverse question types and incorporates metrics to quantify both hallucination rates and reasoning improvements in LLM+KG models. To consider the scenarios in which KGs may contain varying levels of errors, we propose a benchmark variant, OKGQA-P, to assess model performance when the semantics and structure of KGs are deliberately perturbed and contaminated. In this paper, we aims to (1) explore whether KGs can make LLMs more trustworthy in an open-ended setting, and (2) conduct a comparative analysis to shed light on method design. We believe this study can facilitate a more complete performance comparison and encourages continuous improvement in integrating KGs with LLMs to mitigate hallucination, and make LLMs more trustworthy. Code and data are released at https://github.com/Y-Sui/OKGQA.

Figures (8)

• Figure 1: Comparison between open-ended and closed-ended questions over knowledge graphs.
• Figure 2: (left) Dataset statistics and query types, (right) Sub-query type distribution
• Figure 3: Overview of KG-augmented framework.
• Figure 4: Comparison results of different forms of information over different queries.
• Figure 5: Performance Metrics (ATS, SC2D, SD2) vs. Perturbation Level for Different Perturbation Methods.