Rank4Gen: RAG-Preference-Aligned Document Set Selection and Ranking

TL;DR

Rank4Gen tackles the misalignment between traditional relevance-driven ranking and downstream generation quality in retrieval-augmented generation (RAG). It introduces PRISM, a bilingual dataset that provides generator-conditioned supervision for ordered document-set ranking, and a two-stage training pipeline (relevance-oriented SFT followed by direct preference optimization) to learn generator-specific preferences. By modeling both response quality and generator identity, Rank4Gen consistently improves downstream generation quality across five challenging RAG benchmarks and demonstrates robustness to unseen generators. This approach advances practical RAG systems by aligning evidence selection with generation goals, enabling more reliable and generator-aware document ranking.

Abstract

In the RAG paradigm, the information retrieval module provides context for generators by retrieving and ranking multiple documents to support the aggregation of evidence. However, existing ranking models are primarily optimized for query--document relevance, which often misaligns with generators' preferences for evidence selection and citation, limiting their impact on response quality. Moreover, most approaches do not account for preference differences across generators, resulting in unstable cross-generator performance. We propose \textbf{Rank4Gen}, a generator-aware ranker for RAG that targets the goal of \emph{Ranking for Generators}. Rank4Gen introduces two key preference modeling strategies: (1) \textbf{From Ranking Relevance to Response Quality}, which optimizes ranking with respect to downstream response quality rather than query--document relevance; and (2) \textbf{Generator-Specific Preference Modeling}, which conditions a single ranker on different generators to capture their distinct ranking preferences. To enable such modeling, we construct \textbf{PRISM}, a dataset built from multiple open-source corpora and diverse downstream generators. Experiments on five challenging and recent RAG benchmarks demonstrate that RRank4Gen achieves strong and competitive performance for complex evidence composition in RAG.

Figures (16)

• Figure 1: Comparison between traditional ranking and Rank4Gen in RAG. Traditional relevance-based ranking may truncate crucial evidence, while Rank4Gen performs generator-aware document set selection.
• Figure 2: Overview of Rank4Gen. The framework has two stages: (1) PRISM Dataset Construction, which derives generator-aware ranking preferences aligned with downstream response quality; (2) Training and Inference, where Rank4Gen is trained with SFT and DPO on PRISM for generator-aware document set selection ranking.
• Figure 3: Six predefined Arrangement Schemes.
• Figure 4: Impact of document subset composition and ordering on RAG performance across generators.
• Figure A1: Distribution of the number of clusters derived from the fused three-dimensional coordinate representation.