Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SPINACH: SPARQL-Based Information Navigation for Challenging Real-World Questions

Shicheng Liu, Sina J. Semnani, Harold Triedman, Jialiang Xu, Isaac Dan Zhao, Monica S. Lam

TL;DR

This work introduces the SPINACH dataset, an expert-annotated KBQA dataset collected from discussions on Wikidata's "Request a Query"forum with 320 decontextualized question-SPARQL pairs, and introduces an in-context learning KBQA agent that mimics how a human expert would write SPARQLs to handle challenging questions.

Abstract

Large Language Models (LLMs) have led to significant improvements in the Knowledge Base Question Answering (KBQA) task. However, datasets used in KBQA studies do not capture the true complexity of KBQA tasks. They either have simple questions, use synthetically generated logical forms, or are based on small knowledge base (KB) schemas. We introduce the SPINACH dataset, an expert-annotated KBQA dataset collected from discussions on Wikidata's "Request a Query" forum with 320 decontextualized question-SPARQL pairs. The complexity of these in-the-wild queries calls for a KBQA system that can dynamically explore large and often incomplete schemas and reason about them, as it is infeasible to create a comprehensive training dataset. We also introduce an in-context learning KBQA agent, also called SPINACH, that mimics how a human expert would write SPARQLs to handle challenging questions. SPINACH achieves a new state of the art on the QALD-7, QALD-9 Plus and QALD-10 datasets by 31.0%, 27.0%, and 10.0% in $F_1$, respectively, and coming within 1.6% of the fine-tuned LLaMA SOTA model on WikiWebQuestions. On our new SPINACH dataset, the SPINACH agent outperforms all baselines, including the best GPT-4-based KBQA agent, by at least 38.1% in $F_1$.

SPINACH: SPARQL-Based Information Navigation for Challenging Real-World Questions

TL;DR

This work introduces the SPINACH dataset, an expert-annotated KBQA dataset collected from discussions on Wikidata's "Request a Query"forum with 320 decontextualized question-SPARQL pairs, and introduces an in-context learning KBQA agent that mimics how a human expert would write SPARQLs to handle challenging questions.

Abstract

Large Language Models (LLMs) have led to significant improvements in the Knowledge Base Question Answering (KBQA) task. However, datasets used in KBQA studies do not capture the true complexity of KBQA tasks. They either have simple questions, use synthetically generated logical forms, or are based on small knowledge base (KB) schemas. We introduce the SPINACH dataset, an expert-annotated KBQA dataset collected from discussions on Wikidata's "Request a Query" forum with 320 decontextualized question-SPARQL pairs. The complexity of these in-the-wild queries calls for a KBQA system that can dynamically explore large and often incomplete schemas and reason about them, as it is infeasible to create a comprehensive training dataset. We also introduce an in-context learning KBQA agent, also called SPINACH, that mimics how a human expert would write SPARQLs to handle challenging questions. SPINACH achieves a new state of the art on the QALD-7, QALD-9 Plus and QALD-10 datasets by 31.0%, 27.0%, and 10.0% in , respectively, and coming within 1.6% of the fine-tuned LLaMA SOTA model on WikiWebQuestions. On our new SPINACH dataset, the SPINACH agent outperforms all baselines, including the best GPT-4-based KBQA agent, by at least 38.1% in .
Paper Structure (31 sections, 6 equations, 2 figures, 13 tables)

This paper contains 31 sections, 6 equations, 2 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. KBQA Benchmarks
  4. KBQA Approaches
  5. The Spinach Dataset
  6. Choosing and Trimming the Queries
  7. Annotating Natural Questions
  8. The Spinach Agent
  9. Exploration State
  10. Policy
  11. Discussion
  12. Experiments
  13. Evaluation Metrics
  14. Spinach Agent on Prior Datasets
  15. Results on the Spinach Dataset
  16. ...and 16 more sections

Figures (2)

  • Figure 1: The sequence of 13 actions that the Spinach agent takes to answer a sample question from the Spinach validation set. Here, the agent goes through several distinct phases, only with the high-level instruction in Section \ref{['sec:policy']}. Note that every step includes a thought, action and observation, but some are omitted here for brevity. Full version available in Listing \ref{['fig:full_example']}.
  • Figure :