SPARC-RAG: Adaptive Sequential-Parallel Scaling with Context Management for Retrieval-Augmented Generation

TL;DR

This work tackles the inefficiencies and noise that arise when scaling retrieval-augmented generation for multi-hop QA by introducing SPARC-RAG, a multi-agent framework that jointly adapts sequential depth and parallel width under a unified context-management mechanism. It deploys three specialized agents—Query Rewriter, Context Manager, and Answer Evaluator—alongside retrieval and generation components to orchestrate dynamic, memory-aware expansion across branches and rounds. A lightweight, scaling-oriented fine-tuning regime with process-level preference pairs, including a weighted DPO loss, further improves stopping decisions and encourages complementary parallel exploration. Empirical results show that joint depth-width scaling yields superior accuracy-cost trade-offs on single- and multi-hop QA benchmarks, with fine-tuning delivering additional gains in efficiency and robustness. The approach demonstrates that explicit context management and adaptive inference-time scaling can substantially enhance performance for knowledge-intensive tasks while controlling resource usage.

Abstract

Retrieval-Augmented Generation (RAG) grounds large language model outputs in external evidence, but remains challenged on multi-hop question answering that requires long reasoning. Recent works scale RAG at inference time along two complementary dimensions: sequential depth for iterative refinement and parallel width for coverage expansion. However, naive scaling causes context contamination and scaling inefficiency, leading to diminishing or negative returns despite increased computation. To address these limitations, we propose SPARC-RAG, a multi-agent framework that coordinates sequential and parallel inference-time scaling under a unified context management mechanism. SPARC-RAG employs specialized agents that maintain a shared global context and provide explicit control over the scaling process. It generates targeted, complementary sub-queries for each branch to enable diverse parallel exploration, and explicitly regulates exiting decisions based on answer correctness and evidence grounding. To optimize scaling behavior, we further introduce a lightweight fine-tuning method with process-level verifiable preferences, which improves the efficiency of sequential scaling and effectiveness of parallel scaling. Across single- and multi-hop QA benchmarks, SPARC-RAG consistently outperforms previous RAG baselines, yielding an average +6.2 F1 improvement under lower inference cost.

Figures (7)

• Figure 1: Overall SParC-RAG framework. The figure illustrates the interactions, information flow, and decision signals across agents.
• Figure 2: Single left-to-right rollout of the SParC-RAG reasoning process. The Accept (Stop) and Reject (Continue) decisions are produced by the Context Manager after merging the branches.
• Figure 3: F1–cost tradeoff across $(W,D)$ configurations with Qwen2.5-7B-Instruct model. Each panel plots F1 versus tokens per question for one dataset. Curves correspond to widths $W{=}1,2,4$. For $W{=}1$, the sequential depths are $D{=}2,4,6,8,10,12,14$. For $W{=}2$ and $W{=}4$, the depths are $D{=}2,4,6,8$. Along each curve, points are ordered from left to right by increasing $D$. The gold star marks the best untuned $(W,D)$ configuration, and the red diamond denotes our fine-tuned model, which achieves comparable or better F1 at substantially lower token cost.
• Figure 4: Relative performance of different RAG systems using two backbone LLMs across five datasets. Each plot compares Vanilla RAG, the representative parallel (Speculative-RAG) and sequential (DeepNote) RAG scaling methods, and SParC-RAG (ours).
• Figure 5: Answer quality (F1 scores, top row) and retrieval performance (paragraph recall, bottom row) on 2WikiMQA and MuSiQue for $W{=}1,2,4$ using Qwen2.5-7B-Instruct model.