Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic Few-Shot Learning for Knowledge Graph Question Answering

Jacopo D'Abramo, Andrea Zugarini, Paolo Torroni

TL;DR

This paper tackles the KGQA problem by focusing on the query generation step and proposing Dynamic Few-Shot Learning (DFSL), which combines in-context learning with semantic retrieval to dynamically select $k$ similar training examples and enrich prompts. DFSL uses a three-part method: dynamic retrieval of demonstrations, an in-context prompt that incorporates gold entities/relations, and multi-query generation with answer selection to mitigate triple-flip errors and model uncertainty. The approach achieves state-of-the-art or near state-of-the-art results across four KGQA benchmarks (QALD-9 DB, QALD-9 plus, QALD-10, LC-QuAD 2.0) using three backbones (Mixtral, Llama-3, CodeLlama) without fine-tuning, supported by comprehensive ablations. The work highlights the practical impact of dynamic demonstrations and beam-based query generation for scalable, robust KGQA in real-world KG settings, with broad potential for extension to cognate domains such as Text-to-SQL.

Abstract

Large language models present opportunities for innovative Question Answering over Knowledge Graphs (KGQA). However, they are not inherently designed for query generation. To bridge this gap, solutions have been proposed that rely on fine-tuning or ad-hoc architectures, achieving good results but limited out-of-domain distribution generalization. In this study, we introduce a novel approach called Dynamic Few-Shot Learning (DFSL). DFSL integrates the efficiency of in-context learning and semantic similarity and provides a generally applicable solution for KGQA with state-of-the-art performance. We run an extensive evaluation across multiple benchmark datasets and architecture configurations.

Dynamic Few-Shot Learning for Knowledge Graph Question Answering

TL;DR

This paper tackles the KGQA problem by focusing on the query generation step and proposing Dynamic Few-Shot Learning (DFSL), which combines in-context learning with semantic retrieval to dynamically select similar training examples and enrich prompts. DFSL uses a three-part method: dynamic retrieval of demonstrations, an in-context prompt that incorporates gold entities/relations, and multi-query generation with answer selection to mitigate triple-flip errors and model uncertainty. The approach achieves state-of-the-art or near state-of-the-art results across four KGQA benchmarks (QALD-9 DB, QALD-9 plus, QALD-10, LC-QuAD 2.0) using three backbones (Mixtral, Llama-3, CodeLlama) without fine-tuning, supported by comprehensive ablations. The work highlights the practical impact of dynamic demonstrations and beam-based query generation for scalable, robust KGQA in real-world KG settings, with broad potential for extension to cognate domains such as Text-to-SQL.

Abstract

Large language models present opportunities for innovative Question Answering over Knowledge Graphs (KGQA). However, they are not inherently designed for query generation. To bridge this gap, solutions have been proposed that rely on fine-tuning or ad-hoc architectures, achieving good results but limited out-of-domain distribution generalization. In this study, we introduce a novel approach called Dynamic Few-Shot Learning (DFSL). DFSL integrates the efficiency of in-context learning and semantic similarity and provides a generally applicable solution for KGQA with state-of-the-art performance. We run an extensive evaluation across multiple benchmark datasets and architecture configurations.
Paper Structure (40 sections, 3 equations, 4 figures, 7 tables)

This paper contains 40 sections, 3 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Information Retrieval KGQA.
  4. Text-to-SPARQL.
  5. Example Selection in Few-Shot Learning.
  6. Text-to-SQL.
  7. Method
  8. Dynamic Few-Shot Retrieval
  9. In-Context Prompt
  10. Multi-Query Generation
  11. Answer Selection.
  12. Experiments
  13. Datasets
  14. QALD-9 DB.
  15. QALD-9 plus.
  16. ...and 25 more sections

Figures (4)

  • Figure 1: Sketch of DFSL. Given a question, its entities and its relations, $k$-most similar examples are retrieved from a text-to-SPARQL collection $\mathcal{S}$ and injected into the in-context prompt. Then, the LLM generates one or more queries that are all executed by a SPARQL engine. An answer selection strategy identifies which response to pick.
  • Figure 2: Comparison of Embeddings: DFSL (in orange) encoding that incorporates question, entities and relations versus an embedding solely based on the question $q$ (in blue).
  • Figure 3: Impact of the number of in-context examples on the four benchmarks.
  • Figure 4: Examples injected in the Few-shot-learning baseline prompt.