ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA datasets with Large Language Models

TL;DR

Conv-KG-Yarn tackles the need for dynamic, scalable KGQA datasets by leveraging Knowledge Graph structure and LLM-guided data synthesis to generate large-scale, configurable conversational data. The frameworkPipeline integrates KG predicate extraction, LLM-based predicate filtering, related-entity augmentation, and templated question generation across text and voice modalities, producing General and Related dataset variants with millions of facts and thousands of predicates. Rigorous psychometric human evaluation and automated GPT-4 evaluation demonstrate high quality, broad coverage, and strong realism, while revealing how interaction style and linguistic phenomena influence model performance. The work provides a practical, scalable path to train and evaluate evolving conversational AI systems in a rapidly changing information landscape.

Abstract

The rapid advancement of Large Language Models (LLMs) and conversational assistants necessitates dynamic, scalable, and configurable conversational datasets for training and evaluation. These datasets must accommodate diverse user interaction modes, including text and voice, each presenting unique modeling challenges. Knowledge Graphs (KGs), with their structured and evolving nature, offer an ideal foundation for current and precise knowledge. Although human-curated KG-based conversational datasets exist, they struggle to keep pace with the rapidly changing user information needs. We present ConvKGYarn, a scalable method for generating up-to-date and configurable conversational KGQA datasets. Qualitative psychometric analyses confirm our method can generate high-quality datasets rivaling a popular conversational KGQA dataset while offering it at scale and covering a wide range of human-interaction configurations. We showcase its utility by testing LLMs on diverse conversations - exploring model behavior on conversational KGQA sets with different configurations grounded in the same KG fact set. Our results highlight the ability of ConvKGYarn to improve KGQA foundations and evaluate parametric knowledge of LLMs, thus offering a robust solution to the constantly evolving landscape of conversational assistants.

Figures (9)

• Figure 1: The full Conv-KG-Yarn pipeline.
• Figure 2: Prompt for the LLM-based Predicate Selector.
• Figure 3: The prompt used for Synthetic Question Template Generation in the Voice setting.
• Figure 4: Prompt used for Synthetic Question Template Generation in the Text (Search) setting.
• Figure 5: Prompt used for Synthetic Question Template Generation in the Text setting with Relationship Predicates.