PruneRAG: Confidence-Guided Query Decomposition Trees for Efficient Retrieval-Augmented Generation

TL;DR

PruneRAG tackles evidence forgetting and inefficiency in retrieval-augmented generation by introducing a confidence-guided query decomposition tree that adaptively expands, prunes, and aggregates evidence. It unifies answering, decomposition, and fine-grained retrieval, and introduces Evidence Forgetting Rate ($EFR$) as a diagnostic metric. Empirical results across multiple multi-hop QA benchmarks show PruneRAG delivers higher EM/F1 while substantially reducing retrieval cost and $EFR$, including up to 4.9x faster inference. The work demonstrates that principled control over reasoning depth and retrieval granularity is crucial for reliable, scalable evidence utilization in knowledge-intensive tasks.

Abstract

Retrieval-augmented generation (RAG) has become a powerful framework for enhancing large language models in knowledge-intensive and reasoning tasks. However, as reasoning chains deepen or search trees expand, RAG systems often face two persistent failures: evidence forgetting, where retrieved knowledge is not effectively used, and inefficiency, caused by uncontrolled query expansions and redundant retrieval. These issues reveal a critical gap between retrieval and evidence utilization in current RAG architectures. We propose PruneRAG, a confidence-guided query decomposition framework that builds a structured query decomposition tree to perform stable and efficient reasoning. PruneRAG introduces three key mechanisms: adaptive node expansion that regulates tree width and depth, confidence-guided decisions that accept reliable answers and prune uncertain branches, and fine-grained retrieval that extracts entity-level anchors to improve retrieval precision. Together, these components preserve salient evidence throughout multi-hop reasoning while significantly reducing retrieval overhead. To better analyze evidence misuse, we define the Evidence Forgetting Rate as a metric to quantify cases where golden evidence is retrieved but not correctly used. Extensive experiments across various multi-hop QA benchmarks show that PruneRAG achieves superior accuracy and efficiency over state-of-the-art baselines.

Figures (6)

• Figure 1: The left Figure (a) illustrates Multi-turn RAG's sequential decomposition, showing how a low-quality or incorrect answer from an intermediate sub-question propagates, leading to an incorrect final result. The right Figure (b) demonstrates our PruneRAG method. By rejecting the low-quality answer and reflecting to generate a correct, high-quality answer, PruneRAG prevents downward error propagation and successfully arrives at the correct final answer.
• Figure 2: Overall framework of our proposed PruneRAG. The model constructs a query decomposition tree via confidence-guided expansion (Section \ref{['sec:ada']}) and pruning (Section \ref{['sec:pruning']}), performs fine-grained retrieval when decomposition is infeasible (Section \ref{['sec:fine-grained']}), and aggregates intermediate results through a bottom-up backtracing process to generate the final answer (Section \ref{['sec:backtracing']}).
• Figure 3: Retrieval efficiency on HotpotQA, where the x-axis denotes retrieval count, the y-axis denotes golden document recall rate, and bubble size indicates EM.
• Figure 4: Impact of the maximum query tree depth on performance across datasets with varying reasoning complexity.
• Figure 5: Impact of confidence threshold $\tau_A$ on answer accuracy (Exact Match) and average query tree depth across HotpotQA (left) and 2WikiQA (right). The results demonstrate that the threshold range $\tau_A \in [0.9, 0.95]$ consistently achieves strong performance.