A Two-Stage Adaptation of Large Language Models for Text Ranking

TL;DR

This paper tackles the misalignment between pretraining and the text ranking objective in decoder-only LLMs and proposes a two-stage adaptation: continual pre-training (CPT) on a large weakly supervised corpus followed by supervised fine-tuning (SFT) with a full-query ranking objective and auxiliary losses. The approach, TSARankLLM, shows consistent gains over baselines across in-domain (e.g., MS MARCO, DL19, DL20) and out-domain (BEIR) benchmarks and across multiple model families, with ablations confirming the importance of both CPT and SFT components and the auxiliary losses. Key contributions include a novel full-query scoring objective $L_{rank}$, the Differential Penalty $L_{dp}$ to constrain divergence between CPT and SFT, and empirical evidence that stage-wise training improves cross-domain generalization while maintaining efficiency. These findings offer a practical, scalable recipe for deploying decoder-only LLMs in text ranking, balancing performance with computational cost relative to larger, closed models like GPT-4. All mathematical expressions used in the approach are presented in $...$ delimiters.

Abstract

Text ranking is a critical task in information retrieval. Recent advances in pre-trained language models (PLMs), especially large language models (LLMs), present new opportunities for applying them to text ranking. While supervised fine-tuning (SFT) with ranking data has been widely explored to better align PLMs with text ranking goals, previous studies have focused primarily on encoder-only and encoder-decoder PLMs. Research on leveraging decoder-only LLMs for text ranking remains scarce. An exception to this is RankLLaMA, which uses direct SFT to explore LLaMA's potential for text ranking. In this work, we propose a two-stage progressive paradigm to better adapt LLMs to text ranking. First, we conduct continual pre-training (CPT) of LLMs on a large weakly-supervised corpus. Second, we perform SFT, and propose an improved optimization strategy building upon RankLLaMA. Our experimental results on multiple benchmarks show that our approach outperforms previous methods in both in-domain and out-domain scenarios.

Figures (7)

• Figure 1: Misalignment between LLMs (LLaMA) and text ranking objectives: DBLP:conf/emnlp/SachanLJAYPZ22 measures relevance using the probability of generating a query given the document. Unlike ground-truth queries, LLM-generated queries could contain document-irrelevant terms. Such misalignment would lead to suboptimal ranking performance with out-of-the-box LLMs.
• Figure 2: Two-stage adaptation paradigm. The base LLM $\mathcal{M}_{\text{base}}$ turns into an intermediate model $\mathcal{M}_{\text{cpt}}$ after CPT, and then $\mathcal{M}_{\text{cpt}}$ generates the final ranking model $\mathcal{M}_{\text{sft}}$ through SFT.
• Figure 3: Results of TSARankLLM based on LLaMA-7B at various CPT data sizes (millions).
• Figure 4: Model results with different ranking objectives. RankLLaMA is based on the last token, while two variants of our models are based on entire query tokens and do not involve our two auxiliary objectives.
• Figure 5: The perplexity (PPL) comparison of generating positive queries for a given document. "$\Delta$" denotes $\text{PPL}_\text{neg}-\text{PPL}_\text{pos}$, which roughly indicates the model's ranking capability.