Think When Needed: Model-Aware Reasoning Routing for LLM-based Ranking

TL;DR

This paper tackles the high cost and uneven benefits of reasoning in LLM-based ranking by introducing a lightweight, model-aware reasoning router that decides per-instance whether to apply Think or Non-Think using pre-generation signals. It combines ranking-aware features derived from LLM embeddings with model-aware difficulty signals from a masked checklist, and trains a regression-based router to predict the advantage of reasoning, governed by a Pareto-frontier deployment policy. Across three public datasets and multiple open-source LLMs, the router consistently improves ranking utility while reducing token consumption, demonstrating practical efficiency gains and flexible deployment options through various anchors (Knee, Utopia, Epsilon, UMax). This work provides a tangible solution to the accuracy–cost trade-off in LLM-based ranking and lays groundwork for further adaptive, cost-aware inference management in retrieval and recommendation systems.

Abstract

Large language models (LLMs) are increasingly applied to ranking tasks in retrieval and recommendation. Although reasoning prompting can enhance ranking utility, our preliminary exploration reveals that its benefits are inconsistent and come at a substantial computational cost, suggesting that when to reason is as crucial as how to reason. To address this issue, we propose a reasoning routing framework that employs a lightweight, plug-and-play router head to decide whether to use direct inference (Non-Think) or reasoning (Think) for each instance before generation. The router head relies solely on pre-generation signals: i) compact ranking-aware features (e.g., candidate dispersion) and ii) model-aware difficulty signals derived from a diagnostic checklist reflecting the model's estimated need for reasoning. By leveraging these features before generation, the router outputs a controllable token that determines whether to apply the Think mode. Furthermore, the router can adaptively select its operating policy along the validation Pareto frontier during deployment, enabling dynamic allocation of computational resources toward instances most likely to benefit from Think under varying system constraints. Experiments on three public ranking datasets with different scales of open-source LLMs show consistent improvements in ranking utility with reduced token consumption (e.g., +6.3\% NDCG@10 with -49.5\% tokens on MovieLens with Qwen3-4B), demonstrating reasoning routing as a practical solution to the accuracy-efficiency trade-off.

Figures (7)

• Figure 1: Accuracy–efficiency trade-offs across reasoning modes. Left axis shows token count; right axis shows NDCG@10. Modes: Non-Think, Think, Self-Select (the LLM chooses the mode per instance), and Ideal (an oracle selects the better mode for each instance).
• Figure 2: Performance with different reasoning modes on MS-MARCO dataset: (a) Difference of NDCG@10 between Think and Non-Think (denoted by $\Delta \text{NDCG}$). (b) Absolute difference of $\Delta \text{NDCG}$ between Qwen3-14B and Qwen3-4B.
• Figure 3: Prompt template for reasoning routing.
• Figure 4: An overview of our lightweight, model-aware, per-instance router framework.
• Figure 5: Pareto frontier curves on the validation sets of three ranking datasets, showing the trade-off between ranking utility (NDCG@10) and token consumption. Always-Non-Think and Always-Think serve as baselines. Key operating points—including Knee, Utopia, Epsilon, and UMax—are highlighted.