Beyond Contrastive Learning: Synthetic Data Enables List-wise Training with Multiple Levels of Relevance

TL;DR

SyCL introduces a synthetic ranking framework where large language models produce multi-level relevance passages (levels 3,2,1,0) for MS MARCO queries and are trained with a 2-Wasserstein list-wise loss to better capture graded relevance. The approach eliminates real document reliance and demonstrates strong in-domain and out-of-domain ranking gains on MS MARCO and BEIR, with further improvements when real data are incorporated. Key findings show that multi-level synthetic data can outperform binary-label methods, that data quality (not just quantity) matters, and that smaller LLMs can still generate effective training corpora. The work highlights practical benefits for robust IR under distribution shift and niche domains, and outlines avenues for extending relevance granularity and leveraging query collections efficiently.

Abstract

Although synthetic data has changed various aspects of information retrieval (IR) pipelines, the main training paradigm remains: contrastive learning with binary relevance labels, where one positive document is compared against several negatives using the InfoNCE loss. This objective treats all documents that are not explicitly annotated as relevant on an equally negative footing, regardless of their actual degree of relevance, thus missing subtle nuances useful for ranking. To overcome this limitation, in this work, we forgo real documents and annotations and use large language models to directly generate synthetic documents that answer the MS MARCO queries according to several different levels of relevance. We also propose using Wasserstein distance as a more effective loss function for training transformer-based retrievers with graduated relevance labels. Our experiments on MS MARCO and BEIR benchmark show that our proposed approach outperforms conventional training with InfoNCE by a large margin. Without using any real documents, our method significantly improves self-supervised retrievers and is more robust to distribution shift compared to contrastive learning using real data. Our method also successfully integrates existing real data into the synthetic ranking context, further boosting the performance. Overall, we show that generating multi-level ranking contexts is a better approach to synthetic data generation for IR than just generating the standard positive and negative documents.

Figures (5)

• Figure 1: A) Standard contrastive training with real data treats all passages except the explicitly annotated positive passage the same, on a binary basis, regardless of their actual similarity to the given query. It is also vulnerable to false negatives. B) SyCL generates a synthetic multi-level ranking context and trains the model to rank passages based on their degree of relevance to the given query.
• Figure 2: To create a multi-level ranking context for dense retrieval training, we prompt the LLM to sequentially generate four passages with graduated relevance levels for each query. To generate diverse passages, we randomly sample the value of and for each prompt. To avoid easy-to-identify passages, we include the instruction with "*" in the prompt for a random subset of queries. See \ref{['sec:app_prompting']} for details.
• Figure 3: Distribution of the similarity scores between queries and the corresponding synthetic passages for each relevance level.
• Figure 4: Distribution of the top 100 similarity scores across all Dev queries of MS MARCO dataset by models trained with Wasserstein and InfoNCE losses. The model trained with multiple relevance levels learns a more fine-grained notion of relevance.
• Figure 5: Comparison of the distribution of similarity scores for synthetic documents and candidate documents selected for pseudo-labeling by BM25 and E5-Mistral-Instruct. a) For E5-Mistral-Instruct, we only select the the top-4 mined documents. b) For E5-Mistral-Instruct, we select both the top-4 and the the 90$^{th}$ to 95$^{th}$ mined documents for each query.