Dynamic-KGQA: A Scalable Framework for Generating Adaptive Question Answering Datasets

TL;DR

Dynamic-KGQA introduces a scalable, dynamic benchmarking framework for KGQA that generates adaptive QA datasets from a knowledge graph to mitigate data contamination and memorization. It constructs compact, domain-focused seed subgraphs from seed texts, refines them via Steiner-tree extraction, and uses LLMs to generate QA pairs with verification and LLM-as-a-Judge-based annotations. The framework supports randomized QA generation with temperature control and triple reordering, while providing static train/val/test splits for comparability and dynamic variants for continual evaluation. Experiments across multiple LLMs on the YAGO 4.5 KG show that dynamic samples maintain topic consistency and resist contamination, while highlighting the value of KG-grounded reasoning for robust evaluation. Overall, Dynamic-KGQA offers a contamination-resistant, domain-controllable KGQA benchmark with scalable dataset generation and ground-truth reasoning paths, enabling fairer and more extensible evaluation of QA systems.

Abstract

As question answering (QA) systems advance alongside the rapid evolution of foundation models, the need for robust, adaptable, and large-scale evaluation benchmarks becomes increasingly critical. Traditional QA benchmarks are often static and publicly available, making them susceptible to data contamination and memorization by large language models (LLMs). Consequently, static benchmarks may overestimate model generalization and hinder a reliable assessment of real-world performance. In this work, we introduce Dynamic-KGQA, a scalable framework for generating adaptive QA datasets from knowledge graphs (KGs), designed to mitigate memorization risks while maintaining statistical consistency across iterations. Unlike fixed benchmarks, Dynamic-KGQA generates a new dataset variant on every run while preserving the underlying distribution, enabling fair and reproducible evaluations. Furthermore, our framework provides fine-grained control over dataset characteristics, supporting domain-specific and topic-focused QA dataset generation. Additionally, Dynamic-KGQA produces compact, semantically coherent subgraphs that facilitate both training and evaluation of KGQA models, enhancing their ability to leverage structured knowledge effectively. To align with existing evaluation protocols, we also provide static large-scale train/test/validation splits, ensuring comparability with prior methods. By introducing a dynamic, customizable benchmarking paradigm, Dynamic-KGQA enables a more rigorous and adaptable evaluation of QA systems.

Figures (4)

• Figure 1: Overview of Dynamic-KGQA. The framework generates dynamic QA samples while ensuring statistical consistency across runs. Given seed texts, subgraphs are extracted from the KG to ground QA generation. The generated QA pairs are assessed for structural validity, redundancy, and correctness. See prompts P1–P3 in \ref{['app:prompts']} for details.
• Figure 2: Process of constructing a structured subgraph from seed entities. (a) Initial seed entities $\mathcal{S}$ identified from the seed text. (b) Direct edges among $\mathcal{S}$ reveal sparse connectivity. (c) Neighborhood expansion via one-hop triples $\mathcal{T}_{\text{one-hop}}$ yields a denser graph. (d) Steiner tree extraction refines the graph to a minimal, coherent structure while filtering noise.
• Figure 3: Illustration of how a single subgraph can generate multiple QA pairs. Each subfigure represents a different reasoning path within the same knowledge structure to answer diverse but related questions.
• Figure 4: Topic distribution of KGQA datasets.