Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarking

Liangliang Zhang, Zhuorui Jiang, Hongliang Chi, Haoyang Chen, Mohammed Elkoumy, Fali Wang, Qiong Wu, Zhengyi Zhou, Shirui Pan, Suhang Wang, Yao Ma

TL;DR

KGQA benchmarks suffer from quality and evaluation flaws that hinder reliable progress in KG-RAG systems. The authors introduce KGQAGen, an LLM-in-the-loop framework that grounds question generation in Wikidata, iteratively expands grounded subgraphs, and uses SPARQL verification to produce verifiable QA pairs. KGQAGen-10k, a 10k-question benchmark, demonstrates high factual accuracy yet reveals that state-of-the-art models still struggle with grounded multi-hop reasoning, especially under exact-match evaluation. The work advocates rigorous, scalable benchmark construction and presents LASM, a semantic-equality evaluation to better reflect true QA capabilities. Overall, KGQAGen provides a practical path toward more reliable KGQA benchmarking and model development.

Abstract

Knowledge Graph Question Answering (KGQA) systems rely on high-quality benchmarks to evaluate complex multi-hop reasoning. However, despite their widespread use, popular datasets such as WebQSP and CWQ suffer from critical quality issues, including inaccurate or incomplete ground-truth annotations, poorly constructed questions that are ambiguous, trivial, or unanswerable, and outdated or inconsistent knowledge. Through a manual audit of 16 popular KGQA datasets, including WebQSP and CWQ, we find that the average factual correctness rate is only 57 %. To address these issues, we introduce KGQAGen, an LLM-in-the-loop framework that systematically resolves these pitfalls. KGQAGen combines structured knowledge grounding, LLM-guided generation, and symbolic verification to produce challenging and verifiable QA instances. Using KGQAGen, we construct KGQAGen-10k, a ten-thousand scale benchmark grounded in Wikidata, and evaluate a diverse set of KG-RAG models. Experimental results demonstrate that even state-of-the-art systems struggle on this benchmark, highlighting its ability to expose limitations of existing models. Our findings advocate for more rigorous benchmark construction and position KGQAGen as a scalable framework for advancing KGQA evaluation.

Diagnosing and Addressing Pitfalls in KG-RAG Datasets: Toward More Reliable Benchmarking

TL;DR

KGQA benchmarks suffer from quality and evaluation flaws that hinder reliable progress in KG-RAG systems. The authors introduce KGQAGen, an LLM-in-the-loop framework that grounds question generation in Wikidata, iteratively expands grounded subgraphs, and uses SPARQL verification to produce verifiable QA pairs. KGQAGen-10k, a 10k-question benchmark, demonstrates high factual accuracy yet reveals that state-of-the-art models still struggle with grounded multi-hop reasoning, especially under exact-match evaluation. The work advocates rigorous, scalable benchmark construction and presents LASM, a semantic-equality evaluation to better reflect true QA capabilities. Overall, KGQAGen provides a practical path toward more reliable KGQA benchmarking and model development.

Abstract

Knowledge Graph Question Answering (KGQA) systems rely on high-quality benchmarks to evaluate complex multi-hop reasoning. However, despite their widespread use, popular datasets such as WebQSP and CWQ suffer from critical quality issues, including inaccurate or incomplete ground-truth annotations, poorly constructed questions that are ambiguous, trivial, or unanswerable, and outdated or inconsistent knowledge. Through a manual audit of 16 popular KGQA datasets, including WebQSP and CWQ, we find that the average factual correctness rate is only 57 %. To address these issues, we introduce KGQAGen, an LLM-in-the-loop framework that systematically resolves these pitfalls. KGQAGen combines structured knowledge grounding, LLM-guided generation, and symbolic verification to produce challenging and verifiable QA instances. Using KGQAGen, we construct KGQAGen-10k, a ten-thousand scale benchmark grounded in Wikidata, and evaluate a diverse set of KG-RAG models. Experimental results demonstrate that even state-of-the-art systems struggle on this benchmark, highlighting its ability to expose limitations of existing models. Our findings advocate for more rigorous benchmark construction and position KGQAGen as a scalable framework for advancing KGQA evaluation.

Paper Structure

This paper contains 51 sections, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Pitfalls of Existing KGQA Benchmarks
  4. Dataset Quality Issues
  5. Inaccurate Ground Truth Answers.
  6. Low-Quality or Ambiguous Questions
  7. Limitations of Exact-Match Evaluation
  8. KGQAGen: A Framework for Grounded KGQA Dataset Construction
  9. Seed Subgraph Initialization.
  10. Question Generation through Iterative LLM-Guided Subgraph Expansion
  11. LLM-Guided Iterative Subgraph Expansion
  12. Question Generation from the Finalized Subgraph
  13. Answer Validation and Refinement
  14. KGQAGen-10k: A Sample Dataset Generated by KGQAGen
  15. KGQAGen-10k as a Benchmark for KG-RAG Models
  16. ...and 36 more sections

Figures (2)

  • Figure 1: KGQAGen framework.
  • Figure 2: Topic Distribution