Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReasoningRank: Teaching Student Models to Rank through Reasoning-Based Knowledge Distillation

Yuelyu Ji, Zhuochun Li, Rui Meng, Daqing He

TL;DR

Reason-to-Rank (R2R) advances document reranking by jointly producing direct relevance explanations and pairwise comparison explanations, learned via distillation from a large LLM teacher to a compact student. The approach first uses BM25 to gather a candidate pool, then leverages a teacher to output a global ranking plus two types of rationales, which are distilled into a LoRA-fine-tuned student optimized with pairwise, listwise, and generation losses. Across MSMARCO, BEIR, and BRIGHT, R2R achieves competitive or superior reranking performance while enhancing interpretability through dual reasoning, with ablations confirming the additive value of both reasoning types. The work highlights practical benefits for transparency and debugging in IR, while acknowledging limitations such as reliance on expensive teacher models and the need for human evaluation of explanations.

Abstract

Reranking documents based on their relevance to a given query is a critical task in information retrieval. Traditional reranking methods often lack transparency and rely on proprietary models, hindering reproducibility and interpretability. We propose Reason-to-Rank (R2R), a novel open-source reranking approach that enhances transparency by generating two types of reasoning: direct relevance reasoning, which explains how a document addresses the query, and comparison reasoning, which justifies the relevance of one document over another. We leverage large language models (LLMs) as teacher models to generate these explanations and distill this knowledge into smaller, openly available student models. Our student models are trained to generate meaningful reasoning and rerank documents, achieving competitive performance across multiple datasets, including MSMARCO and BRIGHT. Experiments demonstrate that R2R not only improves reranking accuracy but also provides valuable insights into the decision-making process. By offering a structured and interpretable solution with openly accessible resources, R2R aims to bridge the gap between effectiveness and transparency in information retrieval, fostering reproducibility and further research in the field.

ReasoningRank: Teaching Student Models to Rank through Reasoning-Based Knowledge Distillation

TL;DR

Reason-to-Rank (R2R) advances document reranking by jointly producing direct relevance explanations and pairwise comparison explanations, learned via distillation from a large LLM teacher to a compact student. The approach first uses BM25 to gather a candidate pool, then leverages a teacher to output a global ranking plus two types of rationales, which are distilled into a LoRA-fine-tuned student optimized with pairwise, listwise, and generation losses. Across MSMARCO, BEIR, and BRIGHT, R2R achieves competitive or superior reranking performance while enhancing interpretability through dual reasoning, with ablations confirming the additive value of both reasoning types. The work highlights practical benefits for transparency and debugging in IR, while acknowledging limitations such as reliance on expensive teacher models and the need for human evaluation of explanations.

Abstract

Reranking documents based on their relevance to a given query is a critical task in information retrieval. Traditional reranking methods often lack transparency and rely on proprietary models, hindering reproducibility and interpretability. We propose Reason-to-Rank (R2R), a novel open-source reranking approach that enhances transparency by generating two types of reasoning: direct relevance reasoning, which explains how a document addresses the query, and comparison reasoning, which justifies the relevance of one document over another. We leverage large language models (LLMs) as teacher models to generate these explanations and distill this knowledge into smaller, openly available student models. Our student models are trained to generate meaningful reasoning and rerank documents, achieving competitive performance across multiple datasets, including MSMARCO and BRIGHT. Experiments demonstrate that R2R not only improves reranking accuracy but also provides valuable insights into the decision-making process. By offering a structured and interpretable solution with openly accessible resources, R2R aims to bridge the gap between effectiveness and transparency in information retrieval, fostering reproducibility and further research in the field.
Paper Structure (33 sections, 4 equations, 5 figures, 9 tables)

This paper contains 33 sections, 4 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Reranking and Explanation
  4. Direct vs. Comparison Reasoning
  5. Knowledge Distillation in Reranking
  6. Method
  7. Initial Retrieval with BM25.
  8. Teacher Model with Dual Reasoning
  9. Student Model with Knowledge Distillation
  10. Model Overview
  11. Training Objective
  12. Experiments
  13. Datasets and Evaluation Metrics
  14. Evaluation Metrics
  15. Models
  16. ...and 18 more sections

Figures (5)

  • Figure 1: Illustration of the two types of reasoning: direct relevance reasoning provides explicit answers to the query, while comparison reasoning evaluates the relative relevance between documents. The LLM generates these explanations to enhance the interpretability of the ranking process.
  • Figure 2: Overview of direct relevance and comparison reasoning prompts for document ranking. Direct relevance reasoning explains how a document matches a query, while comparison reasoning evaluates the relative relevance between documents.
  • Figure 3: Basic prompt and return type.
  • Figure 4: Overview of the Reason-to-Rank framework. The teacher model generates direct relevance and comparative reasoning, which is used to train student models capable of reproducing the reranked order and generating explanations.
  • Figure 5: Student model performance for different training data sizes on DL19 and DL20.