CompactRAG: Reducing LLM Calls and Token Overhead in Multi-Hop Question Answering

TL;DR

CompactRAG addresses inefficiency in multi-hop QA by decoupling offline corpus preprocessing from online reasoning. It builds an atomic QA knowledge base offline using an LLM and a lightweight online pipeline that resolves queries through dependency-ordered sub-questions, an Answer Extractor, and a Sub-Question Rewriter, with the LLM invoked only twice per query. Experiments on HotpotQA, 2WikiMultiHopQA, and MuSiQue show competitive accuracy while significantly reducing token consumption compared with iterative RAG baselines, validating its cost-efficient design. The offline knowledge base quality, especially when built with a stronger reader (e.g., GPT-4), further boosts online reasoning performance. Overall, CompactRAG demonstrates a practical path to scalable, cost-effective multi-hop reasoning over large knowledge bases.

Abstract

Retrieval-augmented generation (RAG) has become a key paradigm for knowledge-intensive question answering. However, existing multi-hop RAG systems remain inefficient, as they alternate between retrieval and reasoning at each step, resulting in repeated LLM calls, high token consumption, and unstable entity grounding across hops. We propose CompactRAG, a simple yet effective framework that decouples offline corpus restructuring from online reasoning. In the offline stage, an LLM reads the corpus once and converts it into an atomic QA knowledge base, which represents knowledge as minimal, fine-grained question-answer pairs. In the online stage, complex queries are decomposed and carefully rewritten to preserve entity consistency, and are resolved through dense retrieval followed by RoBERTa-based answer extraction. Notably, during inference, the LLM is invoked only twice in total - once for sub-question decomposition and once for final answer synthesis - regardless of the number of reasoning hops. Experiments on HotpotQA, 2WikiMultiHopQA, and MuSiQue demonstrate that CompactRAG achieves competitive accuracy while substantially reducing token consumption compared to iterative RAG baselines, highlighting a cost-efficient and practical approach to multi-hop reasoning over large knowledge corpora. The implementation is available at GitHub.

Figures (5)

• Figure 1: Overview of the Offline Knowledge Construction in CompactRAG. The raw corpus is first processed by an LLM "Reader" that reformulates document content into a set of atomic QA pairs. Each QA pair captures a minimal factual unit, annotated with entity information to ensure semantic completeness and prevent redundancy.
• Figure 2: Overview of the Online Reasoning Pipeline in CompactRAG. The framework begins with query decomposition, where a complex multi-hop question is decomposed into dependency ordered sub-questions. Each sub-question is resolved through iterative retrieval over the atomic QA knowledge base, followed by lightweight answer extraction and question rewriting modules that ensure entity continuity and semantic grounding. Once all sub-questions are resolved, the retrieved QA pairs are aggregated and passed to a final synthesis reasoning step, completing the inference process with only two LLM calls per query.
• Figure 3: Cumulative token consumption on HotpotQA. Although CompactRAG incurs an initial offline cost to construct the atomic QA knowledge base, its cumulative token usage grows slowly and eventually remains well below that of iterative baselines as user queries accumulate.
• Figure 4: Per-query token consumption on HotpotQA. Each point represents one user query (a multi-hop question). The token cost varies with question complexity, leading to oscillations across the curve. Despite this variation, CompactRAG maintains consistently lower per-query consumption than iterative baselines, reflecting its efficiency and stability in online inference.
• Figure 5: Token consumption comparison across additional benchmarks.