Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Beyond Hard Negatives: The Importance of Score Distribution in Knowledge Distillation for Dense Retrieval

Youngjoon Jang, Seongtae Hong, Hyeonseok Moon, Heuiseok Lim

Abstract

Transferring knowledge from a cross-encoder teacher via Knowledge Distillation (KD) has become a standard paradigm for training retrieval models. While existing studies have largely focused on mining hard negatives to improve discrimination, the systematic composition of training data and the resulting teacher score distribution have received relatively less attention. In this work, we highlight that focusing solely on hard negatives prevents the student from learning the comprehensive preference structure of the teacher, potentially hampering generalization. To effectively emulate the teacher score distribution, we propose a Stratified Sampling strategy that uniformly covers the entire score spectrum. Experiments on in-domain and out-of-domain benchmarks confirm that Stratified Sampling, which preserves the variance and entropy of teacher scores, serves as a robust baseline, significantly outperforming top-K and random sampling in diverse settings. These findings suggest that the essence of distillation lies in preserving the diverse range of relative scores perceived by the teacher.

Beyond Hard Negatives: The Importance of Score Distribution in Knowledge Distillation for Dense Retrieval

Abstract

Transferring knowledge from a cross-encoder teacher via Knowledge Distillation (KD) has become a standard paradigm for training retrieval models. While existing studies have largely focused on mining hard negatives to improve discrimination, the systematic composition of training data and the resulting teacher score distribution have received relatively less attention. In this work, we highlight that focusing solely on hard negatives prevents the student from learning the comprehensive preference structure of the teacher, potentially hampering generalization. To effectively emulate the teacher score distribution, we propose a Stratified Sampling strategy that uniformly covers the entire score spectrum. Experiments on in-domain and out-of-domain benchmarks confirm that Stratified Sampling, which preserves the variance and entropy of teacher scores, serves as a robust baseline, significantly outperforming top-K and random sampling in diverse settings. These findings suggest that the essence of distillation lies in preserving the diverse range of relative scores perceived by the teacher.

Paper Structure

This paper contains 16 sections, 3 equations, 2 figures, 3 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORKS
  3. EXPERIMENTAL SETUP
  4. Data Construction and Sampling
  5. Candidate Mining
  6. Sampling
  7. Stratified Sampling Details
  8. Training
  9. Contrastive Learning
  10. Knowledge Distillation
  11. Evaluation
  12. EXPERIMENTAL RESULTS AND ANALYSIS
  13. Main Results
  14. Diversity Statistics
  15. Robustness of Stratified Sampling
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Illustration of candidate selection across different sampling strategies based on the teacher's score distribution.
  • Figure 2: Retrieval performance (nDCG@10) on TREC DL 19 (in-domain) and BEIR (out-of-domain) as the number of sampled candidates $K$ varies ($K\in\{4,8,16\}$). Models are trained with KL-divergence in (a) and with MarginMSE in (b).