Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PRISM: Agentic Retrieval with LLMs for Multi-Hop Question Answering

Md Mahadi Hasan Nahid, Davood Rafiei

TL;DR

PRISM presents a novel agentic retrieval framework for multi-hop QA that splits evidence gathering into precision-focused selection and recall-focused addition, orchestrated by three LLM-powered agents: Question Analyzer, Selector, and Adder. Through iterative cycles (up to $N=3$), PRISM yields compact yet complete evidence sets that significantly improve retrieval quality and translate into stronger end-to-end QA performance across HotpotQA, 2WikiMultiHopQA, MuSiQue, and MultiHopRAG. The approach demonstrates robust gains in both retrieval precision/recall and sentence-level evidence accuracy, while remaining effective across multiple large language model backends. Ablation and error analyses show the importance of decomposition and the precision–recall loop, and the framework maintains generalizability and potential for domain adaptation, despite higher computational cost relative to single-pass retrievers.

Abstract

Retrieval plays a central role in multi-hop question answering (QA), where answering complex questions requires gathering multiple pieces of evidence. We introduce an Agentic Retrieval System that leverages large language models (LLMs) in a structured loop to retrieve relevant evidence with high precision and recall. Our framework consists of three specialized agents: a Question Analyzer that decomposes a multi-hop question into sub-questions, a Selector that identifies the most relevant context for each sub-question (focusing on precision), and an Adder that brings in any missing evidence (focusing on recall). The iterative interaction between Selector and Adder yields a compact yet comprehensive set of supporting passages. In particular, it achieves higher retrieval accuracy while filtering out distracting content, enabling downstream QA models to surpass full-context answer accuracy while relying on significantly less irrelevant information. Experiments on four multi-hop QA benchmarks -- HotpotQA, 2WikiMultiHopQA, MuSiQue, and MultiHopRAG -- demonstrates that our approach consistently outperforms strong baselines.

PRISM: Agentic Retrieval with LLMs for Multi-Hop Question Answering

TL;DR

PRISM presents a novel agentic retrieval framework for multi-hop QA that splits evidence gathering into precision-focused selection and recall-focused addition, orchestrated by three LLM-powered agents: Question Analyzer, Selector, and Adder. Through iterative cycles (up to ), PRISM yields compact yet complete evidence sets that significantly improve retrieval quality and translate into stronger end-to-end QA performance across HotpotQA, 2WikiMultiHopQA, MuSiQue, and MultiHopRAG. The approach demonstrates robust gains in both retrieval precision/recall and sentence-level evidence accuracy, while remaining effective across multiple large language model backends. Ablation and error analyses show the importance of decomposition and the precision–recall loop, and the framework maintains generalizability and potential for domain adaptation, despite higher computational cost relative to single-pass retrievers.

Abstract

Retrieval plays a central role in multi-hop question answering (QA), where answering complex questions requires gathering multiple pieces of evidence. We introduce an Agentic Retrieval System that leverages large language models (LLMs) in a structured loop to retrieve relevant evidence with high precision and recall. Our framework consists of three specialized agents: a Question Analyzer that decomposes a multi-hop question into sub-questions, a Selector that identifies the most relevant context for each sub-question (focusing on precision), and an Adder that brings in any missing evidence (focusing on recall). The iterative interaction between Selector and Adder yields a compact yet comprehensive set of supporting passages. In particular, it achieves higher retrieval accuracy while filtering out distracting content, enabling downstream QA models to surpass full-context answer accuracy while relying on significantly less irrelevant information. Experiments on four multi-hop QA benchmarks -- HotpotQA, 2WikiMultiHopQA, MuSiQue, and MultiHopRAG -- demonstrates that our approach consistently outperforms strong baselines.
Paper Structure (44 sections, 2 figures, 12 tables)

This paper contains 44 sections, 2 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Question Analyzer Agent
  4. Selector Agent
  5. Adder Agent
  6. Iteration and Multi-Hop Handling
  7. Answer Generator Agent
  8. Experimental Setup
  9. Datasets
  10. Baselines and Evaluation Metrics
  11. Baselines.
  12. Evaluation Metrics.
  13. Implementation Details
  14. Results and Discussion
  15. Main Result
  16. ...and 29 more sections

Figures (2)

  • Figure 1: Overview of Agentic Retrieval Framework (PRISM). The complex question is decomposed by the Analyzer into sub-questions, the Selector narrows down relevant evidence for precision, and the Adder expands context for recall. The loop iterates $N$ times to produce a refined evidence set for QA.
  • Figure 2: Passage recall across HotpotQA, 2WikiMultiHopQA, and MuSiQue. Our agentic retrieval framework (PRISM) consistently outperforms OneR and IRCoT trivedi2023ircot, with especially large gains on the challenging MuSiQue benchmark.