Remining Hard Negatives for Generative Pseudo Labeled Domain Adaptation

TL;DR

This work tackles the domain-shift fragility of dense passage retrievers by enhancing Generative Pseudo Labeling (GPL) with Remining of Hard Negatives (R-GPL). By refreshing hard negatives during training using the evolving domain-adapted model, R-GPL yields substantial improvements over GPL across BEIR and LoTTE benchmarks, achieving gains on 13/14 BEIR datasets and 9/12 LoTTE datasets. The approach demonstrates that domain-adapted hard negatives provide stronger training signals and that dynamic remining aligns the dense retriever’s margins with the cross-encoder teacher more effectively. While increasing training complexity, the method preserves inference-time efficiency and offers a practical path to robust zero-shot retrieval in diverse domains.

Abstract

Dense retrievers have demonstrated significant potential for neural information retrieval; however, they exhibit a lack of robustness to domain shifts, thereby limiting their efficacy in zero-shot settings across diverse domains. A state-of-the-art domain adaptation technique is Generative Pseudo Labeling (GPL). GPL uses synthetic query generation and initially mined hard negatives to distill knowledge from cross-encoder to dense retrievers in the target domain. In this paper, we analyze the documents retrieved by the domain-adapted model and discover that these are more relevant to the target queries than those of the non-domain-adapted model. We then propose refreshing the hard-negative index during the knowledge distillation phase to mine better hard negatives. Our remining R-GPL approach boosts ranking performance in 13/14 BEIR datasets and 9/12 LoTTe datasets. Our contributions are (i) analyzing hard negatives returned by domain-adapted and non-domain-adapted models and (ii) applying the GPL training with and without hard-negative re-mining in LoTTE and BEIR datasets.

Figures (6)

• Figure 1: Score distribution of top 100 documents retrieved before and after domain adaptation. Estimated relevance scores based on the teacher model from GPL.
• Figure 2: Predicted relevancy scores for hard negatives retrieved by domain adapted model and hard negative miners. Scores are produced by the teacher model.
• Figure 3: Distillation training loss smoothed with an exponential moving average over length 50. Dashed lines indicate remined hard-negatives.
• Figure 4: Predicted cross encoder (CE) relevance margin between query (Q), relevant document $(D^{+})$, and hard negative document $(D^{-})$. The margin is calculated using CE(Q,$D^{+}$) - CE(Q,$D^{-}$).
• Figure 5: 2D projection of query and document embeddings used in GPL Framework.