R2-Router: A New Paradigm for LLM Routing with Reasoning

TL;DR

The paper tackles the inefficiency of traditional LLM routing that assumes a fixed quality-cost point per model. It introduces R2-Router, which treats output-length as a controllable input, turning each LLM into a quality-cost curve over token budgets and jointly selecting the best (model, budget). Complemented by R2-Bench, a dataset that captures LLM behavior across diverse budgets, the approach enables training a shared encoder with per-budget quality predictors and a reasoning-based routing decision. Empirical results show state-of-the-art performance with 4–5x lower cost, strong generalization to new LLMs and out-of-distribution queries, and easy integration as a plug-in with existing routers. This work reframes routing as deliberate reasoning about resource-constrained generation, offering a practical path to more cost-efficient, scalable LLM deployment.

Abstract

As LLMs proliferate with diverse capabilities and costs, LLM routing has emerged by learning to predict each LLM's quality and cost for a given query, then selecting the one with high quality and low cost. However, existing routers implicitly assume a single fixed quality and cost per LLM for each query, ignoring that the same LLM's quality varies with its output length. This causes routers to exclude powerful LLMs when their estimated cost exceeds the budget, missing the opportunity that these LLMs could still deliver high quality at reduced cost with shorter outputs. To address this, we introduce R2-Router, which treats output length budget as a controllable variable and jointly selects the best LLM and length budget, enforcing the budget via length-constrained instructions. This enables R2-Router to discover that a powerful LLM with constrained output can outperform a weaker LLM at comparable cost-efficient configurations invisible to prior methods. Together with the router framework, we construct R2-Bench, the first routing dataset capturing LLM behavior across diverse output length budgets. Experiments show that R2-Router achieves state-of-the-art performance at 4-5x lower cost compared with existing routers. This work opens a new direction: routing as reasoning, where routers evolve from reactive selectors to deliberate reasoners that explore which LLM to use and at what cost budget.

Figures (8)

• Figure 1: Prior routers take a query and select an LLM based on estimated quality and cost. R2-Router additionally reasons about how quality varies with output length, selecting both the best LLM and an appropriate budget constraint.
• Figure 2: (a) Prior Router: Route on Points. Existing routers profile each LLM as a static point, excluding powerful LLMs like Qwen3-235B when their estimated cost exceeds the budget. (b) R2-Router: Route on Curves. By reasoning about cost-dependent quality, R2-Router transforms each LLM from a point into a curve, discovering that powerful LLMs can deliver superior quality at reduced cost. (c) R2-Router achieves comparable quality at $4-5\times$ lower cost than prior routers.
• Figure 3: Comparison of Oracle performance between R2-Bench and SPROUT\ref{['ft:a']} on three representative LLMs. SPROUT profiles each LLM as a single point (icons), while R2-Bench captures each LLM as a quality-cost curve (solid lines), enabling per-LLM Oracle selection.
• Figure 4: An overview of the proposed framework. (Upper) R2-Bench is built by collecting multiple responses from each LLM under different token budgets (e.g., 10, 50, 500 tokens). An LLM-as-a-judge scores these responses to form quality-cost curves for every (query, LLM) pair. (Lower) R2-Router train a shared encoder with per-LLM, cost-specific quality predictors. At inference time, given a query $x$, a user-defined budget limit $B$, and trade-off coefficient $\lambda$, the router predicts the quality-cost curve for each LLM and selects the optimal (LLM, cost) pair $(M^*, C^*)$ that maximizes $S(x, M, C) = (1-\lambda) \cdot \hat{Q}(x, M, C) - \lambda \cdot C$.
• Figure 5: Deferral curves comparing routing methods. R2-Router achieves consistently higher quality at lower costs than point-based routing methods.

Theorems & Definitions (3)

• Definition 4.1: Reactive Routing
• Definition 4.2: Reasoning-based Routing
• Theorem 4.3: Optimization Dominance