Interactive-KBQA: Multi-Turn Interactions for Knowledge Base Question Answering with Large Language Models

TL;DR

Interactive-KBQA presents a framework that uses a large language model as an agent to perform knowledge base question answering through multi-turn interactions with a KB environment. It introduces three atomic tools to manipulate KBs via SPARQL: SearchNodes, SearchGraphPatterns, and ExecuteSPARQL, enabling the LLM to incrementally derive executable SPARQL expressions S from questions Q, with the KB formalized as $K \in E \times R \times (E \cup L \cup C)$ and the goal of maximizing $p(S|Q,\mathcal{K})$. The approach is supported by a small, high-quality, human-annotated dataset of step-wise reasoning, two exemplars per question type, and a human-in-the-loop process to refine outputs, demonstrating strong performance on WebQSP, CWQ, KQA Pro, and MetaQA in low-resource settings. Empirical results show competitive accuracy with few-shot prompting and open-LM fine-tuning, highlighting the method’s practicality and potential for broader KBQA applications. The work contributes a unified interactive toolset for SPARQL-driven KBQA, a low-resource annotation protocol, and datasets that enable future research into interpretable, multi-turn KB reasoning.

Abstract

This study explores the realm of knowledge base question answering (KBQA). KBQA is considered a challenging task, particularly in parsing intricate questions into executable logical forms. Traditional semantic parsing (SP)-based methods require extensive data annotations, which result in significant costs. Recently, the advent of few-shot in-context learning, powered by large language models (LLMs), has showcased promising capabilities. However, fully leveraging LLMs to parse questions into logical forms in low-resource scenarios poses a substantial challenge. To tackle these hurdles, we introduce Interactive-KBQA, a framework designed to generate logical forms through direct interaction with knowledge bases (KBs). Within this framework, we have developed three generic APIs for KB interaction. For each category of complex question, we devised exemplars to guide LLMs through the reasoning processes. Our method achieves competitive results on the WebQuestionsSP, ComplexWebQuestions, KQA Pro, and MetaQA datasets with a minimal number of examples (shots). Importantly, our approach supports manual intervention, allowing for the iterative refinement of LLM outputs. By annotating a dataset with step-wise reasoning processes, we showcase our model's adaptability and highlight its potential for contributing significant enhancements to the field.

Figures (11)

• Figure 1: Overview of the interactive process.
• Figure 2: An example of the interactive process.
• Figure 3: Instruction text of Freebase.
• Figure 4: Instruction text of Wikidata.
• Figure 5: Instruction text of Movie KG for MetaQA.