Distilling a Small Utility-Based Passage Selector to Enhance Retrieval-Augmented Generation

TL;DR

The paper tackles the cost bottleneck of utility judgments in retrieval-augmented generation by distilling LLM-based utility judgments into compact student models and introducing a forward sliding window for dynamic utility-based passage selection. It trains these students to jointly perform pseudo-answer generation and utility judgments, using a front-to-back process with window size $w$ and stride $s$, based on teacher outputs from Qwen3-32B and data from 100k MS MARCO queries. The resulting models, RankQwen1.7B and UtilityQwen1.7B, demonstrate that utility-based selection outperforms relevance ranking for complex QA tasks like HotpotQA while significantly reducing computational costs, achieving around a 70% efficiency gain. The work provides extensive analyses across BEIR datasets, validates the approach with two generators, and releases the annotated dataset to foster further advancement in utility-based RAG.

Abstract

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating retrieved information. Standard retrieval process prioritized relevance, focusing on topical alignment between queries and passages. In contrast, in RAG, the emphasis has shifted to utility, which considers the usefulness of passages for generating accurate answers. Despite empirical evidence showing the benefits of utility-based retrieval in RAG, the high computational cost of using LLMs for utility judgments limits the number of passages evaluated. This restriction is problematic for complex queries requiring extensive information. To address this, we propose a method to distill the utility judgment capabilities of LLMs into smaller, more efficient models. Our approach focuses on utility-based selection rather than ranking, enabling dynamic passage selection tailored to specific queries without the need for fixed thresholds. We train student models to learn pseudo-answer generation and utility judgments from teacher LLMs, using a sliding window method that dynamically selects useful passages. Our experiments demonstrate that utility-based selection provides a flexible and cost-effective solution for RAG, significantly reducing computational costs while improving answer quality. We present the distillation results using Qwen3-32B as the teacher model for both relevance ranking and utility-based selection, distilled into RankQwen1.7B and UtilityQwen1.7B. Our findings indicate that for complex questions, utility-based selection is more effective than relevance ranking in enhancing answer generation performance. We will release the relevance ranking and utility-based selection annotations for the MS MARCO dataset, supporting further research in this area.

Figures (5)

• Figure 1: Different answer generation performance (generator: Llama-3.1-8B-Instruct) directly with different top-$k$ retrieval results (retriever: BM25).
• Figure 2: The prompt for utility-based selection. The bold is the special part for utility-based selection compared to relevance ranking.
• Figure 3: This diagram depicts an 8 passage selection ($M=8$) process using a sliding window $w=4$, stride $s=2$: (a) relevance ranking and (b) utility-based selection. Greed part means the windows, and the arrow indicates the direction of the window sliding. Passages shown in red are those that have undergone re-ranking or preselected passages. White passages denote unprocessed passages. For utility-based selection, $s$ documents (within the dashed-border box) from the preselected queue are placed into the processing window for utility-based selection. Documents selected within the current window are prepended to the head of the preselected queue.
• Figure 4: Distribution of selected passage count of UtilityQwen$_{1.7B}$ on the NQ and HotpotQA datasets with BM25 retrieval results. "N" means the query numbers of the datasets.
• Figure 5: Distribution of selected passage rank in initial retrieval of UtilityQwen$_{1.7B}$ on the NQ and HotpotQA datasets with initial BM25/BGE retrieval results.