Intermediate Distillation: Data-Efficient Distillation from Black-Box LLMs for Information Retrieval

TL;DR

This work addresses data efficiency in distilling information from black-box LLMs for information retrieval within RAG. It introduces Intermediate Distillation, a two-stage pipeline that first distills LLM ranking into a ranker via ListMLE, then distills the ranker into a retriever by KL-divergence alignment, using only ranking signals rather than LLM probabilities. The approach achieves strong retrieval gains with only around 1k training examples and improves QA performance when integrated into RAG, demonstrating practical efficiency and effectiveness. The study also analyzes data-type and training-set-size effects, showing robust gains even with limited supervision and highlighting the importance of the re-ranking signal over ground-truth placement. Overall, the method offers a scalable, resource-efficient path to leverage black-box LLMs for IR tasks.

Abstract

Recent research has explored distilling knowledge from large language models (LLMs) to optimize retriever models, especially within the retrieval-augmented generation (RAG) framework. However, most existing training methods rely on extracting supervision signals from LLMs' weights or their output probabilities, which is not only resource-intensive but also incompatible with black-box LLMs. In this paper, we introduce \textit{Intermediate Distillation}, a data-efficient knowledge distillation training scheme that treats LLMs as black boxes and distills their knowledge via an innovative LLM-ranker-retriever pipeline, solely using LLMs' ranking generation as the supervision signal. Extensive experiments demonstrate that our proposed method can significantly improve the performance of retriever models with only 1,000 training instances. Moreover, our distilled retriever model significantly boosts performance in question-answering tasks within the RAG framework, demonstrating the potential of LLMs to economically and effectively train smaller models.

Figures (7)

• Figure 1: Previous distillation methods (left) rely on extracting supervision signals from LLM's weights or using LLM's output probabilities to train the retriever model. In contrast, our approach (right) bypasses the need for LLM's likelihood, directly using the LLM's ranking responses as supervision signals.
• Figure 2: The two-stage knowledge distillation process of our proposed Intermediate Distillation scheme. In Stage 1, we use re-ranking order $\pi$ (highlighted in the green background color) as the supervisory signal to train a ranker model. In Stage 2, this distilled ranker unsupervised trains the retriever model to enhance its performance.
• Figure 3: An example of our LLM teacher model's re-ranking process.
• Figure 4: The performance of retriever models across three types of training sets, which vary based on the initial appearance and placement of ground truth in the retrieved subsets.
• Figure 5: The performance of retriever models under different training set size.