LURE-RAG: Lightweight Utility-driven Reranking for Efficient RAG

TL;DR

This work tackles the mismatch between traditional relevance-based retrieval and the actual utility needed to improve downstream generation in retrieval-augmented generation (RAG). It introduces LURE-RAG, a lightweight, black-box retriever framework augmented with a LambdaMART-based reranker trained with listwise utility signals derived from LLM outputs, enabling efficient training and inference while better aligning retrieved documents with generation quality. The authors demonstrate that utility-driven reranking outperforms relevance-based baselines across open-domain QA datasets and multiple LLMs, with a dense variant (UR-RAG) delivering the strongest gains (up to about 3% in accuracy/F1). The approach emphasizes interpretability through feature importance analyses and shows robustness to different LLMs, suggesting practical applicability for real-world RAG systems. Overall, LURE-RAG provides a scalable path to bridge retrieval and generation by optimizing document ordering via task-oriented utility signals.

Abstract

Most conventional Retrieval-Augmented Generation (RAG) pipelines rely on relevance-based retrieval, which often misaligns with utility -- that is, whether the retrieved passages actually improve the quality of the generated text specific to a downstream task such as question answering or query-based summarization. The limitations of existing utility-driven retrieval approaches for RAG are that, firstly, they are resource-intensive typically requiring query encoding, and that secondly, they do not involve listwise ranking loss during training. The latter limitation is particularly critical, as the relative order between documents directly affects generation in RAG. To address this gap, we propose Lightweight Utility-driven Reranking for Efficient RAG (LURE-RAG), a framework that augments any black-box retriever with an efficient LambdaMART-based reranker. Unlike prior methods, LURE-RAG trains the reranker with a listwise ranking loss guided by LLM utility, thereby directly optimizing the ordering of retrieved documents. Experiments on two standard datasets demonstrate that LURE-RAG achieves competitive performance, reaching 97-98% of the state-of-the-art dense neural baseline, while remaining efficient in both training and inference. Moreover, its dense variant, UR-RAG, significantly outperforms the best existing baseline by up to 3%.

Figures (2)

• Figure 1: Schematic diagram of LURE-RAG workflow. ① For the given query, documents are retrieved using a (black-box) retriever. ② Each of these documents is used as a context for the query and the prompts are constructed. These prompts are fed into a generator one by one to get the LLM predictions. ③ LLM's posteriors are used to compute the utility of each document. The documents are then sorted based on the utility scores and only top-$k$ of them are retained. These form the training instances for training a reranker. ④ LambdaMART is trained using thus obtained training data.
• Figure 2: Feature importance scores of the top-8 features obtained from the trained lambdaMART model for LURE-RAG.