Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distillation versus Contrastive Learning: How to Train Your Rerankers

Zhichao Xu, Zhiqi Huang, Shengyao Zhuang, Vivek Srikumar

TL;DR

This work compares two dominant training paradigms for cross-encoder rerankers—contrastive learning ($L_{InfoNCE}$) and knowledge distillation (KL divergence between softened teacher and student outputs with temperature $T$)—across multiple model sizes and architectures. Distilling from a strong 7B teacher generally yields superior in-domain and out-of-domain ranking on MS MARCO, DL19/DL20, and BEIR, though gains diminish when the teacher and student have equal capacity. The experiments, including a data-decoupled RLHN setup, show KD's robustness and scalability advantages, while noting CL can still be competitive when no strong teacher is available. The findings provide practical guidance: use KD to train smaller rerankers if a powerful teacher exists; otherwise, CL remains a viable baseline; code is released to support reproducibility.

Abstract

Training effective text rerankers is crucial for information retrieval. Two strategies are widely used: contrastive learning (optimizing directly on ground-truth labels) and knowledge distillation (transferring knowledge from a larger reranker). While both have been studied extensively, a clear comparison of their effectiveness for training cross-encoder rerankers under practical conditions is needed. This paper empirically compares these strategies by training rerankers of different sizes (0.5B, 1.5B, 3B, 7B) and architectures (Transformer, Recurrent) using both methods on the same data, with a strong contrastive learning model acting as the distillation teacher. Our results show that knowledge distillation generally yields better in-domain and out-of-domain ranking performance than contrastive learning when distilling from a more performant teacher model. This finding is consistent across student model sizes and architectures. However, distilling from a teacher of the same capacity does not provide the same advantage, particularly for out-of-domain tasks. These findings offer practical guidance for choosing a training strategy based on available teacher models. We recommend using knowledge distillation to train smaller rerankers if a larger, more performant teacher is accessible; in its absence, contrastive learning remains a robust baseline. Our code implementation is made available to facilitate reproducbility.

Distillation versus Contrastive Learning: How to Train Your Rerankers

TL;DR

This work compares two dominant training paradigms for cross-encoder rerankers—contrastive learning () and knowledge distillation (KL divergence between softened teacher and student outputs with temperature )—across multiple model sizes and architectures. Distilling from a strong 7B teacher generally yields superior in-domain and out-of-domain ranking on MS MARCO, DL19/DL20, and BEIR, though gains diminish when the teacher and student have equal capacity. The experiments, including a data-decoupled RLHN setup, show KD's robustness and scalability advantages, while noting CL can still be competitive when no strong teacher is available. The findings provide practical guidance: use KD to train smaller rerankers if a powerful teacher exists; otherwise, CL remains a viable baseline; code is released to support reproducibility.

Abstract

Training effective text rerankers is crucial for information retrieval. Two strategies are widely used: contrastive learning (optimizing directly on ground-truth labels) and knowledge distillation (transferring knowledge from a larger reranker). While both have been studied extensively, a clear comparison of their effectiveness for training cross-encoder rerankers under practical conditions is needed. This paper empirically compares these strategies by training rerankers of different sizes (0.5B, 1.5B, 3B, 7B) and architectures (Transformer, Recurrent) using both methods on the same data, with a strong contrastive learning model acting as the distillation teacher. Our results show that knowledge distillation generally yields better in-domain and out-of-domain ranking performance than contrastive learning when distilling from a more performant teacher model. This finding is consistent across student model sizes and architectures. However, distilling from a teacher of the same capacity does not provide the same advantage, particularly for out-of-domain tasks. These findings offer practical guidance for choosing a training strategy based on available teacher models. We recommend using knowledge distillation to train smaller rerankers if a larger, more performant teacher is accessible; in its absence, contrastive learning remains a robust baseline. Our code implementation is made available to facilitate reproducbility.

Paper Structure

This paper contains 24 sections, 5 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. The Text Reranking Problem
  5. Training with Contrastive Learning
  6. Training with Knowledge Distillation
  7. Experimental Setup
  8. Training setup.
  9. Evaluation setup.
  10. Models used.
  11. Baselines.
  12. Implementation details.
  13. Results and Analysis
  14. Main Results
  15. In-domain results.
  16. ...and 9 more sections

Figures (2)

  • Figure 1: Scaling trend for contrastive learning vs knowledge distillation, for passage reranking. Left figure shows TREC DL19+20 results while right figure shows BEIR results averaged over 12 datasets (excluding Quora). We skip results of RankQwen-7B-KD model.
  • Figure 2: Dataset scaling trend for CL versus KD. Left figure shows average results for 7 in-domain datasets from BEIR benchmark, while the right figure shows average results for 7 out-of-domain datasets.