Don't Retrieve, Generate: Prompting LLMs for Synthetic Training Data in Dense Retrieval
Aarush Sinha
TL;DR
The paper investigates whether synthetic hard negatives generated by large language models can replace corpus-based hard-negative mining for training dense retrievers. It builds a corpus-free pipeline that prompts multiple LLMs to produce hard negatives and evaluates training DistilBERT with these signals against BM25 and cross-encoder baselines across the BEIR benchmark. Results show that traditional corpus-based supervision outperforms LLM-generated negatives, and that increasing LLM size does not guarantee better training signals; in some cases smaller models perform best. Additionally, naïve concatenation of synthetic and retrieved negatives yields mixed results, underscoring the need for principled integration strategies between generative and retrieval-based supervision.
Abstract
Training effective dense retrieval models typically relies on hard negative (HN) examples mined from large document corpora using methods such as BM25 or cross-encoders, which require full corpus access and expensive index construction. We propose generating synthetic hard negatives directly from a provided query and positive passage, using Large Language Models(LLMs). We fine-tune DistilBERT using synthetic negatives generated by four state-of-the-art LLMs ranging from 4B to 30B parameters (Qwen3, LLaMA3, Phi4) and evaluate performance across 10 BEIR benchmark datasets. Contrary to the prevailing assumption that stronger generative models yield better synthetic data, find that our generative pipeline consistently underperforms traditional corpus-based mining strategies (BM25 and Cross-Encoder). Furthermore, we observe that scaling the generator model does not monotonically improve retrieval performance and find that the 14B parameter model outperforms the 30B model and in some settings it is the worst performing.