REARANK: Reasoning Re-ranking Agent via Reinforcement Learning

TL;DR

REARANK introduces a reasoning listwise reranking agent trained with reinforcement learning to explicitly reason before reordering candidate passages. By leveraging a data-efficient augmentation pipeline (179 annotated queries) and a GRPO-based RL objective, Rearank achieves substantial gains over baselines and matches or surpasses GPT-4 on several benchmarks, including reasoning-intensive BRIGHT tasks, while remaining compact for local deployment. The approach yields interpretable reasoning in outputs and demonstrates transferability of improved reasoning to mathematical reasoning tasks, underscoring the practical impact of integrating explicit reasoning into IR reranking. Overall, the work advances data-efficient, explainable LLM reranking with strong robustness in both in-domain and out-of-domain settings.

Abstract

We present REARANK, a large language model (LLM)-based listwise reasoning reranking agent. REARANK explicitly reasons before reranking, significantly improving both performance and interpretability. Leveraging reinforcement learning and data augmentation, REARANK achieves substantial improvements over baseline models across popular information retrieval benchmarks, notably requiring only 179 annotated samples. Built on top of Qwen2.5-7B, our REARANK-7B demonstrates performance comparable to GPT-4 on both in-domain and out-of-domain benchmarks and even surpasses GPT-4 on reasoning-intensive BRIGHT benchmarks. These results underscore the effectiveness of our approach and highlight how reinforcement learning can enhance LLM reasoning capabilities in reranking.

Figures (8)

• Figure 1: (Top) Average rerank results on popular benchmarks (over BM25 top 100), the performance improves with RL training; (Bottom) Rearank inference example. The agent provides the reasoning and final ranking of all passages, unlike current agents rankgptpradeep2023rankzephyr that only output the final answer.
• Figure 2: Listwise vs. Setwise Reranking. Setwise reranking yields binary scores (0 or 1); listwise reranking offers richer, continuous scores between 0 and 1.
• Figure 3: Pipeline of the proposed GRPO-based RL framework for listwise passage reranking. Training utilizes data generated by sampling multiple passage sets per query and evaluating them with consistent relevance judgments.
• Figure 4: (Top) Reward evolving curve (Bottom) Response length curve.
• Figure 5: Reasoning patterns: Before- vs. After-RL training under identical prompt and query.