RRADistill: Distilling LLMs' Passage Ranking Ability for Long-Tail Queries Document Re-Ranking on a Search Engine

TL;DR

This research proposes an efficient label generation pipeline and novel sLLM training methods for both encoder and decoder models, and introduces an encoder-based method using a Term Control Layer to capture term matching signals and a decoder-based model with a ranking layer for enhanced understanding.

Abstract

Large Language Models (LLMs) excel at understanding the semantic relationships between queries and documents, even with lengthy and complex long-tail queries. These queries are challenging for feedback-based rankings due to sparse user engagement and limited feedback, making LLMs' ranking ability highly valuable. However, the large size and slow inference of LLMs necessitate the development of smaller, more efficient models (sLLMs). Recently, integrating ranking label generation into distillation techniques has become crucial, but existing methods underutilize LLMs' capabilities and are cumbersome. Our research, RRADistill: Re-Ranking Ability Distillation, propose an efficient label generation pipeline and novel sLLM training methods for both encoder and decoder models. We introduce an encoder-based method using a Term Control Layer to capture term matching signals and a decoder-based model with a ranking layer for enhanced understanding. A/B testing on a Korean-based search platform, validates the effectiveness of our approach in improving re-ranking for long-tail queries.

Figures (9)

• Figure 1: An example of zero-shot inference by HyperCLOVA X (HCX) on retrieved documents. The red box shows HCX's ranked output; the blue box shows excluded documents. Irrelevant parts of excluded documents are highlighted in yellow. The original was in Korean, but translated to English.
• Figure 2: The overview of our label generation pipeline. Negatives are randomly selected from documents totally unrelated to the query.
• Figure 3: RRA-BERT: The [SEP] token distinguishes the query and the document. The Term Control Layer can be omitted during inference.
• Figure 4: RRA-GPT: The special token here is <|Response|>. The label and reasoning generation can be omitted during inference.
• Figure 5: Training data format: Replace the red sections$\{\{query\}\}, \{\{snippet\text{ }text\}\}, \{\{label\}\}, \{\{reasoning\}\}$ as needed.