Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

REAPER: Reasoning based Retrieval Planning for Complex RAG Systems

Ashutosh Joshi, Sheikh Muhammad Sarwar, Samarth Varshney, Sreyashi Nag, Shrivats Agrawal, Juhi Naik

TL;DR

REAPER introduces a reasoning-based planner to generate end-to-end retrieval plans for complex RAG queries, using a compact LLM to determine which tools to call, in what order, and with which arguments. It achieves high plan accuracy and data efficiency, reducing latency by operating with a smaller model than traditional agent-based or multi-agent approaches. The authors demonstrate data-efficiency (6K in-domain queries vs 150K for baselines), scalable expansion to new retrieval sources with minimal data (~286 queries), and substantial latency improvements (~207 ms vs 2 s). The approach generalizes to unseen tools and plan structures without explicit retraining, offering practical benefits for scalable, latency-sensitive conversational systems such as shopping assistants.

Abstract

Complex dialog systems often use retrieved evidence to facilitate factual responses. Such RAG (Retrieval Augmented Generation) systems retrieve from massive heterogeneous data stores that are usually architected as multiple indexes or APIs instead of a single monolithic source. For a given query, relevant evidence needs to be retrieved from one or a small subset of possible retrieval sources. Complex queries can even require multi-step retrieval. For example, a conversational agent on a retail site answering customer questions about past orders will need to retrieve the appropriate customer order first and then the evidence relevant to the customer's question in the context of the ordered product. Most RAG Agents handle such Chain-of-Thought (CoT) tasks by interleaving reasoning and retrieval steps. However, each reasoning step directly adds to the latency of the system. For large models this latency cost is significant -- in the order of multiple seconds. Multi-agent systems may classify the query to a single Agent associated with a retrieval source, though this means that a (small) classification model dictates the performance of a large language model. In this work we present REAPER (REAsoning-based PlannER) - an LLM based planner to generate retrieval plans in conversational systems. We show significant gains in latency over Agent-based systems and are able to scale easily to new and unseen use cases as compared to classification-based planning. Though our method can be applied to any RAG system, we show our results in the context of a conversational shopping assistant.

REAPER: Reasoning based Retrieval Planning for Complex RAG Systems

TL;DR

REAPER introduces a reasoning-based planner to generate end-to-end retrieval plans for complex RAG queries, using a compact LLM to determine which tools to call, in what order, and with which arguments. It achieves high plan accuracy and data efficiency, reducing latency by operating with a smaller model than traditional agent-based or multi-agent approaches. The authors demonstrate data-efficiency (6K in-domain queries vs 150K for baselines), scalable expansion to new retrieval sources with minimal data (~286 queries), and substantial latency improvements (~207 ms vs 2 s). The approach generalizes to unseen tools and plan structures without explicit retraining, offering practical benefits for scalable, latency-sensitive conversational systems such as shopping assistants.

Abstract

Complex dialog systems often use retrieved evidence to facilitate factual responses. Such RAG (Retrieval Augmented Generation) systems retrieve from massive heterogeneous data stores that are usually architected as multiple indexes or APIs instead of a single monolithic source. For a given query, relevant evidence needs to be retrieved from one or a small subset of possible retrieval sources. Complex queries can even require multi-step retrieval. For example, a conversational agent on a retail site answering customer questions about past orders will need to retrieve the appropriate customer order first and then the evidence relevant to the customer's question in the context of the ordered product. Most RAG Agents handle such Chain-of-Thought (CoT) tasks by interleaving reasoning and retrieval steps. However, each reasoning step directly adds to the latency of the system. For large models this latency cost is significant -- in the order of multiple seconds. Multi-agent systems may classify the query to a single Agent associated with a retrieval source, though this means that a (small) classification model dictates the performance of a large language model. In this work we present REAPER (REAsoning-based PlannER) - an LLM based planner to generate retrieval plans in conversational systems. We show significant gains in latency over Agent-based systems and are able to scale easily to new and unseen use cases as compared to classification-based planning. Though our method can be applied to any RAG system, we show our results in the context of a conversational shopping assistant.
Paper Structure (7 sections, 3 figures)

This paper contains 7 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Our Contribution
  3. Literature Review
  4. Problem Statement
  5. Baselines
  6. REAPER
  7. REAPER Prompt Design

Figures (3)

  • Figure 1: Traditional RAG systems rely on retrieving evidence in parallel from one or more sources. Conversational shopping can include features like personalization (questions about past purchases, preferences, subscriptions, etc), shopping recommendations, and more that require multi-step retrieval. These use cases can be complex enough that either an Agent is required to identify the steps, or retrieval needs its own CoT planner. We introduce REAPER for CoT retrieval planning
  • Figure 2: Example: REAPER plans. Note that REAPER is able to incorporate context (second plan) and generate multi-step retrieval plans when necessary (last plan)
  • Figure 3: Example: Mistral plans. For simple queries (first and third query) Mistral produces the right plans. However, for multi-step retrieval Mistral goes beyond just retrieval evidence generation, hallucinating steps like Compare (second plan) and Extract estimated delivery time (fourth plan).