Lightweight and Direct Document Relevance Optimization for Generative Information Retrieval

TL;DR

This paper tackles the misalignment between token-level docid generation and document-level relevance in Generative Information Retrieval (GenIR). It introduces Direct Document Relevance Optimization (DDRO), a direct Learning-to-Rank (L2R) framework that pairs relevance signals with constrained, end-to-end docid generation, avoiding explicit reward models and reinforcement learning. DDRO employs two docid designs (Title+URL and Product Quantization) and a two-stage training pipeline: supervised fine-tuning (SFT) followed by direct L2R optimization using relevance feedback, formalized through a pairwise objective that leverages implicit rewards without RL. Experimental results on MS MARCO Document Ranking and Natural Questions show DDRO achieving strong improvements over RL-based GenRRL and competitive performance against established baselines, demonstrating a lightweight yet effective path to improved GenIR ranking across datasets and docid designs.

Abstract

Generative information retrieval (GenIR) is a promising neural retrieval paradigm that formulates document retrieval as a document identifier (docid) generation task, allowing for end-to-end optimization toward a unified global retrieval objective. However, existing GenIR models suffer from token-level misalignment, where models trained to predict the next token often fail to capture document-level relevance effectively. While reinforcement learning-based methods, such as reinforcement learning from relevance feedback (RLRF), aim to address this misalignment through reward modeling, they introduce significant complexity, requiring the optimization of an auxiliary reward function followed by reinforcement fine-tuning, which is computationally expensive and often unstable. To address these challenges, we propose direct document relevance optimization (DDRO), which aligns token-level docid generation with document-level relevance estimation through direct optimization via pairwise ranking, eliminating the need for explicit reward modeling and reinforcement learning. Experimental results on benchmark datasets, including MS MARCO document and Natural Questions, show that DDRO outperforms reinforcement learning-based methods, achieving a 7.4% improvement in MRR@10 for MS MARCO and a 19.9% improvement for Natural Questions. These findings highlight DDRO's potential to enhance retrieval effectiveness with a simplified optimization approach. By framing alignment as a direct optimization problem, DDRO simplifies the ranking optimization pipeline of GenIR models while offering a viable alternative to reinforcement learning-based methods.

Figures (4)

• Figure 1: The proposed workflow comprises three key stages: (1) Construction of document identifiers (docids) including URL/title, domain, and product quantization codes; (2) Supervised fine-tuning of the retrieval model ${\pi_\theta^\mathit{ref}}$ using diverse data pairs; and (3) Freezing the trained reference policy model ${\pi_\theta^\mathit{ref}}$ and performing direct learning-to-rank (L2R) optimization on a policy model $\pi_\theta$.
• Figure 2: Gradient for direct learning-to-rank optimization using relevance feedback.
• Figure 3: Architecture of the DDRO model, which fine-tunes the retrieval model through direct learning-to-rank (L2R) optimization using relevance feedback. Unlike GenRRL zhou-etal-2023-enhancing-generative, DDRO directly optimizes with relevance judgment data, avoiding reinforcement learning, explicit reward modeling, and extensive hyperparameter tuning. For clarity, the model ${\pi_\theta^\mathit{ref}}$ from the SFT phase is referred to as ${\pi^\mathit{ref}}$, with its parameters frozen during this phase.
• Figure 4: Effect of KL constraint strength ($\beta$) on DDRO performance. A moderate value ($\beta = 0.4$) yields the best MRR@10, while under- or over-regularization degrades performance.