Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Triad: A Framework Leveraging a Multi-Role LLM-based Agent to Solve Knowledge Base Question Answering

Chang Zong, Yuchen Yan, Weiming Lu, Jian Shao, Eliot Huang, Heng Chang, Yueting Zhuang

TL;DR

This work tackles knowledge-base question answering (KBQA) under limited task-specific training data by introducing Triad, a unified framework that employs a single LLM-based agent playing three distinct roles. The architecture distributes KBQA subtasks across four phases—question parsing, URI linking, query construction, and answer generation—through a cooperative interplay among a Generalist (G-Agent), a Decision-Maker (D-Agent), and an Advisor (A-Agent). Empirical results on LC-QuAD, YAGO-QA, and QALD-9 show Triad achieving competitive or superior performance relative to state-of-the-art KBQA systems and pure LLM baselines, with notable gains on LC-QuAD and YAGO-QA (F1 improvements of 11.8% and 20.7%). The analysis highlights the importance of role-specific components, hyperparameter tuning, and explicit knowledge memory, while also outlining limitations such as handling more complex multi-hop questions and the potential cost benefits from fewer-shot prompts.

Abstract

Recent progress with LLM-based agents has shown promising results across various tasks. However, their use in answering questions from knowledge bases remains largely unexplored. Implementing a KBQA system using traditional methods is challenging due to the shortage of task-specific training data and the complexity of creating task-focused model structures. In this paper, we present Triad, a unified framework that utilizes an LLM-based agent with three roles for KBQA tasks. The agent is assigned three roles to tackle different KBQA subtasks: agent as a generalist for mastering various subtasks, as a decision maker for the selection of candidates, and as an advisor for answering questions with knowledge. Our KBQA framework is executed in four phases, involving the collaboration of the agent's multiple roles. We evaluated the performance of our framework using three benchmark datasets, and the results show that our framework outperforms state-of-the-art systems on the LC-QuAD and YAGO-QA benchmarks, yielding F1 scores of 11.8% and 20.7%, respectively.

Triad: A Framework Leveraging a Multi-Role LLM-based Agent to Solve Knowledge Base Question Answering

TL;DR

This work tackles knowledge-base question answering (KBQA) under limited task-specific training data by introducing Triad, a unified framework that employs a single LLM-based agent playing three distinct roles. The architecture distributes KBQA subtasks across four phases—question parsing, URI linking, query construction, and answer generation—through a cooperative interplay among a Generalist (G-Agent), a Decision-Maker (D-Agent), and an Advisor (A-Agent). Empirical results on LC-QuAD, YAGO-QA, and QALD-9 show Triad achieving competitive or superior performance relative to state-of-the-art KBQA systems and pure LLM baselines, with notable gains on LC-QuAD and YAGO-QA (F1 improvements of 11.8% and 20.7%). The analysis highlights the importance of role-specific components, hyperparameter tuning, and explicit knowledge memory, while also outlining limitations such as handling more complex multi-hop questions and the potential cost benefits from fewer-shot prompts.

Abstract

Recent progress with LLM-based agents has shown promising results across various tasks. However, their use in answering questions from knowledge bases remains largely unexplored. Implementing a KBQA system using traditional methods is challenging due to the shortage of task-specific training data and the complexity of creating task-focused model structures. In this paper, we present Triad, a unified framework that utilizes an LLM-based agent with three roles for KBQA tasks. The agent is assigned three roles to tackle different KBQA subtasks: agent as a generalist for mastering various subtasks, as a decision maker for the selection of candidates, and as an advisor for answering questions with knowledge. Our KBQA framework is executed in four phases, involving the collaboration of the agent's multiple roles. We evaluated the performance of our framework using three benchmark datasets, and the results show that our framework outperforms state-of-the-art systems on the LC-QuAD and YAGO-QA benchmarks, yielding F1 scores of 11.8% and 20.7%, respectively.
Paper Structure (54 sections, 8 equations, 3 figures, 6 tables)

This paper contains 54 sections, 8 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Phases of KBQA
  4. Question parsing
  5. URI linking
  6. Query construction
  7. Answer generation
  8. Roles of LLM-based Agent
  9. Agent as a generalist
  10. Agent as a decision-maker
  11. Agent as an advisor
  12. Task Formulation
  13. Triad Framework
  14. G-Agent as a Generalized Solver
  15. Triplet mention extraction:
  16. ...and 39 more sections

Figures (3)

  • Figure 1: A system with multiple roles who focus on sub-problems of each phase to solve a complex task.
  • Figure 2: Our Triad framework leverages an LLM-based agent with three different roles including a generalist, a decision-maker, and an advisor to cooperatively handle a series of subtasks in the four phases of a KBQA process.
  • Figure 3: Response time of traditional KBQA systems and Triad on three datasets. Each bar shows average response time of a particular phase of KBQA.