Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaling Laws for Reranking in Information Retrieval

Rahul Seetharaman, Aman Bansal, Hamed Zamani, Kaustubh Dhole

TL;DR

This work presents the first systematic study of scaling laws for rerankers by analyzing performance across model sizes and data budgets for three popular paradigms: pointwise, pairwise, and listwise reranking and demonstrates that performance follows a predictable power law.

Abstract

Scaling laws have been observed across a wide range of tasks, such as natural language generation and dense retrieval, where performance follows predictable patterns as model size, data, and compute grow. However, these scaling laws are insufficient for understanding the scaling behavior of multi-stage retrieval systems, which typically include a reranking stage. In large-scale multi-stage retrieval systems, reranking is the final and most influential step before presenting a ranked list of items to the end user. In this work, we present the first systematic study of scaling laws for rerankers by analyzing performance across model sizes and data budgets for three popular paradigms: pointwise, pairwise, and listwise reranking. Using a detailed case study with cross-encoder rerankers, we demonstrate that performance follows a predictable power law. This regularity allows us to accurately forecast the performance of larger models for some metrics more than others using smaller-scale experiments, offering a robust methodology for saving significant computational resources. For example, we accurately estimate the NDCG of a 1B-parameter model by training and evaluating only smaller models (up to 400M parameters), in both in-domain as well as out-of-domain settings. Our experiments encompass span several loss functions, models and metrics and demonstrate that downstream metrics like NDCG, MAP (Mean Avg Precision) show reliable scaling behavior and can be forecasted accurately at scale, while highlighting the limitations of metrics like Contrastive Entropy and MRR (Mean Reciprocal Rank) which do not follow predictable scaling behavior in all instances. Our results establish scaling principles for reranking and provide actionable insights for building industrial-grade retrieval systems.

Scaling Laws for Reranking in Information Retrieval

TL;DR

This work presents the first systematic study of scaling laws for rerankers by analyzing performance across model sizes and data budgets for three popular paradigms: pointwise, pairwise, and listwise reranking and demonstrates that performance follows a predictable power law.

Abstract

Scaling laws have been observed across a wide range of tasks, such as natural language generation and dense retrieval, where performance follows predictable patterns as model size, data, and compute grow. However, these scaling laws are insufficient for understanding the scaling behavior of multi-stage retrieval systems, which typically include a reranking stage. In large-scale multi-stage retrieval systems, reranking is the final and most influential step before presenting a ranked list of items to the end user. In this work, we present the first systematic study of scaling laws for rerankers by analyzing performance across model sizes and data budgets for three popular paradigms: pointwise, pairwise, and listwise reranking. Using a detailed case study with cross-encoder rerankers, we demonstrate that performance follows a predictable power law. This regularity allows us to accurately forecast the performance of larger models for some metrics more than others using smaller-scale experiments, offering a robust methodology for saving significant computational resources. For example, we accurately estimate the NDCG of a 1B-parameter model by training and evaluating only smaller models (up to 400M parameters), in both in-domain as well as out-of-domain settings. Our experiments encompass span several loss functions, models and metrics and demonstrate that downstream metrics like NDCG, MAP (Mean Avg Precision) show reliable scaling behavior and can be forecasted accurately at scale, while highlighting the limitations of metrics like Contrastive Entropy and MRR (Mean Reciprocal Rank) which do not follow predictable scaling behavior in all instances. Our results establish scaling principles for reranking and provide actionable insights for building industrial-grade retrieval systems.
Paper Structure (31 sections, 5 equations, 5 figures, 4 tables)

This paper contains 31 sections, 5 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Multi-Stage Retrieval and Reranking.
  4. Scaling Laws and Forecasting in Machine Learning.
  5. Scaling Laws in Retrieval.
  6. Reranking Objectives and the Missing Scaling Picture.
  7. Reranking Paradigms
  8. A Framework for Reranker Scaling Laws
  9. Training, Fitting, and Evaluation Protocol
  10. Evaluation Metrics
  11. Scaling
  12. Model Size Scaling
  13. Data Size Scaling
  14. Joint Scaling
  15. Experiments
  16. ...and 16 more sections

Figures (5)

  • Figure 1: Scaling behavior of NDCG@10 (top row) and Contrastive Entropy (bottom row) under model, data, and joint scaling.
  • Figure 2: Model scaling trends on TREC DL '19 across NDCG@10, MAP, and MRR.
  • Figure 3: MAP scaling trends (MSMARCO dev set) showing predictable scaling behavior.
  • Figure 4: MRR scaling trends (MSMARCO dev set) showing predictable scaling behavior.
  • Figure 5: MRR scaling trends (TREC DL '20) - listwise and pairwise scale differently than pointwise.