Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Few-shot In-context Learning for Knowledge Base Question Answering

Tianle Li, Xueguang Ma, Alex Zhuang, Yu Gu, Yu Su, Wenhu Chen

TL;DR

The paper tackles KBQA under low data and schema heterogeneity by introducing KB-BINDER, a training-free, few-shot in-context learning framework. It splits the task into drafting executable logical forms with Codex and grounding those drafts to real KB entities and relations via two binders, with self-consistency and retrieval-augmented prompting variants. Across GrailQA, GraphQA, WebQSP, and MetaQA, KB-BINDER achieves strong performance, surpassing some fully supervised baselines on GraphQA and 3-hop MetaQA and remaining competitive on other datasets. This approach offers a practical, general baseline for low-resource KBQA and demonstrates the potential of LLM based in-context reasoning for complex knowledge graphs.

Abstract

Question answering over knowledge bases is considered a difficult problem due to the challenge of generalizing to a wide variety of possible natural language questions. Additionally, the heterogeneity of knowledge base schema items between different knowledge bases often necessitates specialized training for different knowledge base question-answering (KBQA) datasets. To handle questions over diverse KBQA datasets with a unified training-free framework, we propose KB-BINDER, which for the first time enables few-shot in-context learning over KBQA tasks. Firstly, KB-BINDER leverages large language models like Codex to generate logical forms as the draft for a specific question by imitating a few demonstrations. Secondly, KB-BINDER grounds on the knowledge base to bind the generated draft to an executable one with BM25 score matching. The experimental results on four public heterogeneous KBQA datasets show that KB-BINDER can achieve a strong performance with only a few in-context demonstrations. Especially on GraphQA and 3-hop MetaQA, KB-BINDER can even outperform the state-of-the-art trained models. On GrailQA and WebQSP, our model is also on par with other fully-trained models. We believe KB-BINDER can serve as an important baseline for future research. Our code is available at https://github.com/ltl3A87/KB-BINDER.

Few-shot In-context Learning for Knowledge Base Question Answering

TL;DR

The paper tackles KBQA under low data and schema heterogeneity by introducing KB-BINDER, a training-free, few-shot in-context learning framework. It splits the task into drafting executable logical forms with Codex and grounding those drafts to real KB entities and relations via two binders, with self-consistency and retrieval-augmented prompting variants. Across GrailQA, GraphQA, WebQSP, and MetaQA, KB-BINDER achieves strong performance, surpassing some fully supervised baselines on GraphQA and 3-hop MetaQA and remaining competitive on other datasets. This approach offers a practical, general baseline for low-resource KBQA and demonstrates the potential of LLM based in-context reasoning for complex knowledge graphs.

Abstract

Question answering over knowledge bases is considered a difficult problem due to the challenge of generalizing to a wide variety of possible natural language questions. Additionally, the heterogeneity of knowledge base schema items between different knowledge bases often necessitates specialized training for different knowledge base question-answering (KBQA) datasets. To handle questions over diverse KBQA datasets with a unified training-free framework, we propose KB-BINDER, which for the first time enables few-shot in-context learning over KBQA tasks. Firstly, KB-BINDER leverages large language models like Codex to generate logical forms as the draft for a specific question by imitating a few demonstrations. Secondly, KB-BINDER grounds on the knowledge base to bind the generated draft to an executable one with BM25 score matching. The experimental results on four public heterogeneous KBQA datasets show that KB-BINDER can achieve a strong performance with only a few in-context demonstrations. Especially on GraphQA and 3-hop MetaQA, KB-BINDER can even outperform the state-of-the-art trained models. On GrailQA and WebQSP, our model is also on par with other fully-trained models. We believe KB-BINDER can serve as an important baseline for future research. Our code is available at https://github.com/ltl3A87/KB-BINDER.
Paper Structure (21 sections, 5 figures, 6 tables)

This paper contains 21 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Drafts Generator
  5. Knowledge Base Binder
  6. Entity Binder
  7. Relation Binder
  8. Majority Vote
  9. Retrieved Exemplars
  10. Experiment
  11. Datasets
  12. Baselines
  13. Implementation Details
  14. Main Result
  15. KB-BINDER Results
  16. ...and 6 more sections

Figures (5)

  • Figure 1: Overview of KB-BINDER pipeline. There are two primary stages in our method: 1) Generate the drafts as preliminary logical forms; 2) Bind the drafts to the executable ones with entity and relation binders grounded on the knowledge base. The final answer can be obtained after the execution of the final candidates.
  • Figure 2: KB-BINDER framework: Given a question, the LLM will first generate its corresponding preliminary logical forms as the drafts, imitating the exemplary demonstration. Then the entity and relation binders will operate on the drafts to ground the entities and relations on KB respectively, which produces the final candidates.
  • Figure 3: KB-BINDER coverage and EM scores trend with shot number.
  • Figure 4: KB-BINDER coverage and EM scores trend with top K self-consistency.
  • Figure 5: Positive and negative examples generated by KB-BINDER.