Don't "Overthink" Passage Reranking: Is Reasoning Truly Necessary?
Nour Jedidi, Yung-Sung Chuang, James Glass, Jimmy Lin
TL;DR
This study questions the necessity of explicit chain-of-thought reasoning in LLM-based passage reranking by comparing StandardRR, ReasonRR, and ReasonRR-NoReason under identical training. Across in-domain and out-of-domain datasets, StandardRR consistently outperforms ReasonRR, with Reasoning often degrading performance as model scale grows. Even when ReasonRR's reasoning is forced off (ReasonRR-NoReason), gains appear, particularly on BRIGHT at larger scales, suggesting reasoning biases the model toward polarized relevance and away from partial relevance. The authors discuss the importance of partial relevance modeling and propose future directions, such as calibrating scores or training with graded relevance, while advocating stronger baselines and careful baselining when evaluating reasoning-based rerankers.
Abstract
With the growing success of reasoning models across complex natural language tasks, researchers in the Information Retrieval (IR) community have begun exploring how similar reasoning capabilities can be integrated into passage rerankers built on Large Language Models (LLMs). These methods typically employ an LLM to produce an explicit, step-by-step reasoning process before arriving at a final relevance prediction. But, does reasoning actually improve reranking accuracy? In this paper, we dive deeper into this question, studying the impact of the reasoning process by comparing reasoning-based pointwise rerankers (ReasonRR) to standard, non-reasoning pointwise rerankers (StandardRR) under identical training conditions, and observe that StandardRR generally outperforms ReasonRR. Building on this observation, we then study the importance of reasoning to ReasonRR by disabling its reasoning process (ReasonRR-NoReason), and find that ReasonRR-NoReason is surprisingly more effective than ReasonRR. Examining the cause of this result, our findings reveal that reasoning-based rerankers are limited by the LLM's reasoning process, which pushes it toward polarized relevance scores and thus fails to consider the partial relevance of passages, a key factor for the accuracy of pointwise rerankers.