SGSH: Stimulate Large Language Models with Skeleton Heuristics for Knowledge Base Question Generation

TL;DR

This work tackles knowledge-base question generation by leveraging large language models more effectively through skeleton heuristics. It introduces SGSH, a framework combining a skeleton generator (trained with learnable prompts via an automatic data-construction pipeline) and a frozen GPT-3.5 model, using skeleton injection and skeleton-aware in-context learning to steer question generation toward ground-truth questions. Empirical results on WebQuestions and PathQuestions show SGSH achieving state-of-the-art performance, with notable improvements even when training data for skeleton generation is limited. The approach also yields positive downstream effects on KBQA data augmentation and human evaluation demonstrates superior fluency, relevance, and diversity. This skeleton-guided prompting strategy offers a practical path to harnessing LLM knowledge for structured NLP tasks like KBQG, with potential applicability to related prompt-tuning and data-construction challenges.

Abstract

Knowledge base question generation (KBQG) aims to generate natural language questions from a set of triplet facts extracted from KB. Existing methods have significantly boosted the performance of KBQG via pre-trained language models (PLMs) thanks to the richly endowed semantic knowledge. With the advance of pre-training techniques, large language models (LLMs) (e.g., GPT-3.5) undoubtedly possess much more semantic knowledge. Therefore, how to effectively organize and exploit the abundant knowledge for KBQG becomes the focus of our study. In this work, we propose SGSH--a simple and effective framework to Stimulate GPT-3.5 with Skeleton Heuristics to enhance KBQG. The framework incorporates "skeleton heuristics", which provides more fine-grained guidance associated with each input to stimulate LLMs to generate optimal questions, encompassing essential elements like the question phrase and the auxiliary verb.More specifically, we devise an automatic data construction strategy leveraging ChatGPT to construct a skeleton training dataset, based on which we employ a soft prompting approach to train a BART model dedicated to generating the skeleton associated with each input. Subsequently, skeleton heuristics are encoded into the prompt to incentivize GPT-3.5 to generate desired questions. Extensive experiments demonstrate that SGSH derives the new state-of-the-art performance on the KBQG tasks.

Figures (8)

• Figure 1: Performance comparison between three advanced methods for KBQG under different numbers of in-context examples on the WebQuestions dataset. The methods include the state-of-the-art PLM-based method DSM (yellow), text-davinci-003 (green), and text-davinci-003 with skeleton heuristics (blue).
• Figure 2: Overview of our SGSH framework, which consists of a PLM-based skeleton generator and a frozen GPT-3.5 model. The skeleton generator, optimized by the learnable prompting strategy, generates the skeleton for each test input. Subsequently, GPT-3.5 leverages skeleton heuristics through skeleton injection and skeleton-aware in-context learning to generate the desired question.
• Figure 3: Illustration of the automatic training data construction strategy. We use ChatGPT as an automatic scorer to rate each skeleton generated by the rule-based and ChatGPT-based methods on a scale of 0 to 1.
• Figure 4: A skeleton heuristics-enhanced prompt for Davinci003 on KBQG.
• Figure 5: A ChatGPT prompt for generating skeletons.