Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Autoregressive Ranking: Bridging the Gap Between Dual and Cross Encoders

Benjamin Rozonoyer, Chong You, Michael Boratko, Himanshu Jain, Nilesh Gupta, Srinadh Bhojanapalli, Andrew McCallum, Felix Yu

TL;DR

The paper introduces Autoregressive Ranking (ARR), a unified ranking framework that generates docIDs token-by-token conditioned on a query to rank documents via beam search, offering a potential replacement for the traditional DE/CE pipeline. It provides a theoretical capacity analysis showing that DEs require embedding dimensions that scale with the candidate set, while ARR can rank arbitrarily many documents with a fixed hidden size, and introduces SToICaL, a rank-aware loss combining item- and token-level supervision. The authors develop a prefix-tree–based target distribution and rank-aware reweighting to train ARR effectively, and validate the approach on WordNet and ESCI datasets, demonstrating reduced generation of invalid docIDs and improved ranking metrics compared to DEs, with competitive performance relative to CEs. The results suggest ARR can exploit in-context reasoning and decoding strategies to achieve efficient, accurate ranking without separate indexing, with practical implications for scalable IR and LLM-based retrieval. Overall, the work formalizes ARR’s expressive capacity, proposes principled rank-aware training, and shows empirical gains in constrained generation and ranking quality.

Abstract

Dual and cross encoders have long been mainstays of information retrieval (IR), but are being challenged by the emergent capabilities of LLMs. An LLM-based approach we term pointwise generative ranking - generating tokens the length of a single docID as opposed to a list in order to enable ranking via beam search - combines efficiency and expressivity benefits while leveraging the in-context capabilities of Causal Transformers. Although there is ample evidence to suggest that pretrained LLMs are well-suited for ranking, we find that the vast majority of LLM-based approaches rely on next-token prediction, a loss function which is fundamentally rank-agnostic (and especially so with pointwise supervision). In this paper, we first prove that the expressivity of pointwise generative ranking with multi-token docIDs is superior to that of dual encoders. We then propose SToICaL - a Simple Token-Item Calibrated Loss - which can incorporate rank-aware supervision at both the item and token levels within the pointwise setup. We run a suite of experiments on ranking tasks derived from WordNet (Fellbaum, 1998) and ESCI (Reddy et al., arXiv:2206.06588). Two variants of SToICaL successfully suppress the probability of invalid docID generations and improve on common ranking metrics beyond top-1 retrieval.

Autoregressive Ranking: Bridging the Gap Between Dual and Cross Encoders

TL;DR

The paper introduces Autoregressive Ranking (ARR), a unified ranking framework that generates docIDs token-by-token conditioned on a query to rank documents via beam search, offering a potential replacement for the traditional DE/CE pipeline. It provides a theoretical capacity analysis showing that DEs require embedding dimensions that scale with the candidate set, while ARR can rank arbitrarily many documents with a fixed hidden size, and introduces SToICaL, a rank-aware loss combining item- and token-level supervision. The authors develop a prefix-tree–based target distribution and rank-aware reweighting to train ARR effectively, and validate the approach on WordNet and ESCI datasets, demonstrating reduced generation of invalid docIDs and improved ranking metrics compared to DEs, with competitive performance relative to CEs. The results suggest ARR can exploit in-context reasoning and decoding strategies to achieve efficient, accurate ranking without separate indexing, with practical implications for scalable IR and LLM-based retrieval. Overall, the work formalizes ARR’s expressive capacity, proposes principled rank-aware training, and shows empirical gains in constrained generation and ranking quality.

Abstract

Dual and cross encoders have long been mainstays of information retrieval (IR), but are being challenged by the emergent capabilities of LLMs. An LLM-based approach we term pointwise generative ranking - generating tokens the length of a single docID as opposed to a list in order to enable ranking via beam search - combines efficiency and expressivity benefits while leveraging the in-context capabilities of Causal Transformers. Although there is ample evidence to suggest that pretrained LLMs are well-suited for ranking, we find that the vast majority of LLM-based approaches rely on next-token prediction, a loss function which is fundamentally rank-agnostic (and especially so with pointwise supervision). In this paper, we first prove that the expressivity of pointwise generative ranking with multi-token docIDs is superior to that of dual encoders. We then propose SToICaL - a Simple Token-Item Calibrated Loss - which can incorporate rank-aware supervision at both the item and token levels within the pointwise setup. We run a suite of experiments on ranking tasks derived from WordNet (Fellbaum, 1998) and ESCI (Reddy et al., arXiv:2206.06588). Two variants of SToICaL successfully suppress the probability of invalid docID generations and improve on common ranking metrics beyond top-1 retrieval.
Paper Structure (34 sections, 4 theorems, 15 equations, 5 figures, 3 tables)

This paper contains 34 sections, 4 theorems, 15 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Paper Contributions.
  3. Related Work
  4. Types of Losses for Retrieval and Ranking
  5. Generative Reranking
  6. Parametric Generative Retrieval and DocID Creation
  7. The Capacity for Ranking
  8. Autoregressive Ranking (ARR).
  9. Scoring-Based Ranking Architecture
  10. Ranking via Dual Encoders
  11. Ranking via (Pointwise) Generative Rankers
  12. A Rank-aware Training Loss
  13. Generalized Loss Formulation for Generative Ranking
  14. Rank-Aware Item-Level Reweighting
  15. Prefix Tree for Rank-Aware Token-Level Target Distributions $\mathbf{y}(r,t)$
  16. ...and 19 more sections

Key Result

Theorem 3.1

Let $f_\text{DE}(q, d; \theta)$ be a dual encoder architecture defined in eq:dual-encoder-ranking-architecture with some embedding dimension $n$. Then there does not exist a $\theta$ such that $f_\text{DE}(q, d; \theta)$ solves a complete ranking task when where $k := |\mathscr{D}|$.

Figures (5)

  • Figure 1: Illustration of different architectures for ranking. Red and green circles represent query and document tokens, respectively. (a) DEs are efficient but have limited expressive power. (b) CEs produce more accurate relevance scores but have high computational cost. Hence, the standard practice is a two-stage pipeline where a DE selects a candidate set and a CE reranks those candidates. (c) ARR holds the promise of a unified model for ranking that replaces the DE/CE pipeline.
  • Figure 2: Illustration of a prefix-tree (\ref{['fig:prefix-tree']}); target distributions derived from the prefix tree via marginalization are shown in (\ref{['fig:marg-over-trie']}); one-hot supervision as shown in (\ref{['fig:one-hots']}) can only provide signal for a single document at a time (i.e., top-1).
  • Figure 3: Results on WordNet with Dual Encoders (DEs) and Cross Encoders (CEs). (a) Comparion of Pointwise generative ranking with DEs of varying embedding dimension $n \in \{4, 8, 16, 32\}$ and a 4-layer CE. (b, c) Effect of $\alpha$ in the reweighting function $\lambda(r)=\frac{1}{r^\alpha}$ of the training loss for DEs and CEs, respectively.
  • Figure :
  • Figure :

Theorems & Definitions (10)

  • definition 3.1: Ranking Task
  • definition 3.2: Scoring-based Ranking Architecture
  • definition 3.3: Complete Ranking Task
  • Theorem 3.1: Insufficiency of Dual Encoders for Complete Ranking
  • proof
  • Proposition 3.1
  • proof
  • Corollary 3.1
  • Proposition 3.2
  • proof