Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Spider4SPARQL: A Complex Benchmark for Evaluating Knowledge Graph Question Answering Systems

Catherine Kosten, Philippe Cudré-Mauroux, Kurt Stockinger

TL;DR

Spider4SPARQL is introduced -a new SPARQL benchmark dataset featuring 9,693 previously existing manually generated NL questions and 4,721 unique, novel, and complex SPARQL queries of varying complexity, demonstrating that Spider4SPARQL is a challenging benchmark for future research.

Abstract

With the recent spike in the number and availability of Large Language Models (LLMs), it has become increasingly important to provide large and realistic benchmarks for evaluating Knowledge Graph Question Answering (KGQA) systems. So far the majority of benchmarks rely on pattern-based SPARQL query generation approaches. The subsequent natural language (NL) question generation is conducted through crowdsourcing or other automated methods, such as rule-based paraphrasing or NL question templates. Although some of these datasets are of considerable size, their pitfall lies in their pattern-based generation approaches, which do not always generalize well to the vague and linguistically diverse questions asked by humans in real-world contexts. In this paper, we introduce Spider4SPARQL - a new SPARQL benchmark dataset featuring 9,693 previously existing manually generated NL questions and 4,721 unique, novel, and complex SPARQL queries of varying complexity. In addition to the NL/SPARQL pairs, we also provide their corresponding 166 knowledge graphs and ontologies, which cover 138 different domains. Our complex benchmark enables novel ways of evaluating the strengths and weaknesses of modern KGQA systems. We evaluate the system with state-of-the-art KGQA systems as well as LLMs, which achieve only up to 45\% execution accuracy, demonstrating that Spider4SPARQL is a challenging benchmark for future research.

Spider4SPARQL: A Complex Benchmark for Evaluating Knowledge Graph Question Answering Systems

TL;DR

Spider4SPARQL is introduced -a new SPARQL benchmark dataset featuring 9,693 previously existing manually generated NL questions and 4,721 unique, novel, and complex SPARQL queries of varying complexity, demonstrating that Spider4SPARQL is a challenging benchmark for future research.

Abstract

With the recent spike in the number and availability of Large Language Models (LLMs), it has become increasingly important to provide large and realistic benchmarks for evaluating Knowledge Graph Question Answering (KGQA) systems. So far the majority of benchmarks rely on pattern-based SPARQL query generation approaches. The subsequent natural language (NL) question generation is conducted through crowdsourcing or other automated methods, such as rule-based paraphrasing or NL question templates. Although some of these datasets are of considerable size, their pitfall lies in their pattern-based generation approaches, which do not always generalize well to the vague and linguistically diverse questions asked by humans in real-world contexts. In this paper, we introduce Spider4SPARQL - a new SPARQL benchmark dataset featuring 9,693 previously existing manually generated NL questions and 4,721 unique, novel, and complex SPARQL queries of varying complexity. In addition to the NL/SPARQL pairs, we also provide their corresponding 166 knowledge graphs and ontologies, which cover 138 different domains. Our complex benchmark enables novel ways of evaluating the strengths and weaknesses of modern KGQA systems. We evaluate the system with state-of-the-art KGQA systems as well as LLMs, which achieve only up to 45\% execution accuracy, demonstrating that Spider4SPARQL is a challenging benchmark for future research.
Paper Structure (32 sections, 5 figures, 5 tables)

This paper contains 32 sections, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work and Analysis of Existing Benchmarks
  3. Knowledge Graphs
  4. Benchmark Datasets for Evaluating KGQA Systems
  5. Spider4SPARQL - A New Benchmark for Evaluating Text-to-SQL Systems
  6. Constructing Knowledge Graphs based on Relational Databases
  7. Applying Direct Mapping between DBs and KGs
  8. Converting SQLite Databases to PostgreSQL
  9. Converting SQL Queries to SPARQL
  10. Converting NL to SemQL
  11. Converting SemQL to SPARQL
  12. Challenges of SQL to SPARQL Conversion
  13. Aggregation Queries
  14. Set Operation Queries
  15. SPARQL Query Translation Quality Control
  16. ...and 17 more sections

Figures (5)

  • Figure 1: Schema of the flight_2 database from the Spider dataset. Note that there are no primary/foreign keys between the tables flights and airlines.
  • Figure 2: A portion of a knowledge graph showing the source and destination airports of flights. Classes are highlighted in grey, data property edges are labelled in green and object property edges in blue. Literals correspond to the base data of a relational database and are shown in red.
  • Figure 3: Conceptual framework of virtual knowledge graphs. Various data sources are exposed via ontology mappings either as a virtual or a materialized knowledge graph.
  • Figure 4: The figure shows the execution accuracy for different query categories. The numbers on top of the bars show the number of sample queries per category.
  • Figure 5: The figure shows the execution accuracy for easy, medium, hard and extra hard queries, across the baseline systems. The numbers on top of the bars indicate how many queries are in each category.