Enhancing Transformer-Based Rerankers with Synthetic Data and LLM-Based Supervision

TL;DR

The paper tackles the challenge of domain-specific document reranking when labeled query-document pairs are scarce. It introduces a query-free fine-tuning pipeline that uses LLMs to generate synthetic queries from domain corpora and to label positives and hard negatives via an LLM-based classifier, training a compact cross-encoder with Localized Contrastive Estimation (LCE). By leveraging LLMs for data generation and supervision rather than inference, the approach reduces computational cost while achieving strong in-domain gains and robust out-domain generalization on MedQuAD and MS MARCO subsets, as evidenced by metrics such as $MAP@10$, $MRR@10$, and $NDCG@10$. The method enables scalable domain adaptation for retrieval-augmented generation and other IR tasks without manual labeling, with future work exploring reinforcement learning for data generation and multilingual/knowledge-graph enhancements.

Abstract

Effective document reranking is essential for improving search relevance across diverse applications. While Large Language Models (LLMs) excel at reranking due to their deep semantic understanding and reasoning, their high computational cost makes them impractical for many real-world deployments. Fine-tuning smaller, task-specific models is a more efficient alternative but typically depends on scarce, manually labeled data. To overcome this, we propose a novel pipeline that eliminates the need for human-labeled query-document pairs. Our method uses LLMs to generate synthetic queries from domain-specific corpora and employs an LLM-based classifier to label positive and hard-negative pairs. This synthetic dataset is then used to fine-tune a smaller transformer model with contrastive learning using Localized Contrastive Estimation (LCE) loss. Experiments on the MedQuAD dataset show that our approach significantly boosts in-domain performance and generalizes well to out-of-domain tasks. By using LLMs for data generation and supervision rather than inference, we reduce computational costs while maintaining strong reranking capabilities.

Figures (4)

• Figure 1: Synthetic Dataset Generation Workflow: (i) Generate a synthetic query $q$ from a given text corpus using a large language model (LLM). (ii) Retrieve the most relevant documents $D_q$ for $q$ from the corpus using a bi-encoder model. (iii) Evaluate each query-document pair in $D_q$ using an LLM-based relevance scoring function $f_{RC}$. (iv) Classify documents as relevant (positive) or irrelevant (negative) based on a predefined relevance threshold.
• Figure 2: Mean Average Precision ($MAP@10$), Mean Reciprocal Rank ($MRR@10$), and Normalized Discounted Cumulative Gain ($NDCG@10$) for the in-domain dataset, computed per epoch up to the 10th epoch. The training dataset sizes range from 100 to 1000 in increments of 100. These results demonstrate the effect of dataset size on model performance throughout the fine-tuning process.
• Figure 3: Mean Average Precision ($MAP@10$), Mean Reciprocal Rank ($MRR@10$), and Normalized Discounted Cumulative Gain ($NDCG@10$) for the out-domain dataset, evaluated per epoch up to the 10th epoch. The training dataset sizes vary from 100 to 1000 in increments of 100, providing insights into the model's generalization performance during fine-tuning.
• Figure 4: Impact of training dataset size on performance metrics for both in-domain and out-of-domain datasets. The figure shows the average performance improvement (before vs. after training), computed across all epochs, with standard deviations representing variability.