Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Models Under SCOPE: Scalable and Controllable Routing via Pre-hoc Reasoning

Qi Cao, Shuhao Zhang, Ruizhe Zhou, Ruiyi Zhang, Peijia Qin, Pengtao Xie

TL;DR

This work tackles efficient and controllable routing among large language models by moving beyond fixed model selections to a pre-hoc, reasoning-based estimator that predicts both accuracy and cost. Scope leverages retrieval-augmented fingerprinting of model behavior on an anchor set to generalize to unseen models without retraining, and it optimizes a budget-aware utility that balances accuracy and expense. The framework is trained in two stages—SFT with hindsight distillation and reinforcement learning via GRPO—and augmented with anchor-based calibration to improve robustness. Experiments show Scope can outperform individual models and existing routers in both accuracy and cost across diverse regimes and unseen models, enabling flexible test-time scaling and practical deployment. The open-source plan aims to provide datasets, fingerprints, and full pipeline tooling to foster adoption and further research into budget-aware, reasoning-enabled routing.

Abstract

Model routing chooses which language model to use for each query. By sending easy queries to cheaper models and hard queries to stronger ones, it can significantly reduce inference cost while maintaining high accuracy. However, most existing routers treat this as a fixed choice among a small set of models, which makes them hard to adapt to new models or changing budget constraints. In this paper, we propose SCOPE (Scalable and Controllable Outcome Performance Estimator), a routing framework that goes beyond model selection by predicting their cost and performance. Trained with reinforcement learning, SCOPE makes reasoning-based predictions by retrieving how models behave on similar problems, rather than relying on fixed model names, enabling it to work with new, unseen models. Moreover, by explicitly predicting how accurate and how expensive a model will be, it turns routing into a dynamic decision problem, allowing users to easily control the trade-off between accuracy and cost. Experiments show that SCOPE is more than just a cost-saving tool. It flexibly adapts to user needs: it can boost accuracy by up to 25.7% when performance is the priority, or cut costs by up to 95.1% when efficiency matters most.

Models Under SCOPE: Scalable and Controllable Routing via Pre-hoc Reasoning

TL;DR

This work tackles efficient and controllable routing among large language models by moving beyond fixed model selections to a pre-hoc, reasoning-based estimator that predicts both accuracy and cost. Scope leverages retrieval-augmented fingerprinting of model behavior on an anchor set to generalize to unseen models without retraining, and it optimizes a budget-aware utility that balances accuracy and expense. The framework is trained in two stages—SFT with hindsight distillation and reinforcement learning via GRPO—and augmented with anchor-based calibration to improve robustness. Experiments show Scope can outperform individual models and existing routers in both accuracy and cost across diverse regimes and unseen models, enabling flexible test-time scaling and practical deployment. The open-source plan aims to provide datasets, fingerprints, and full pipeline tooling to foster adoption and further research into budget-aware, reasoning-enabled routing.

Abstract

Model routing chooses which language model to use for each query. By sending easy queries to cheaper models and hard queries to stronger ones, it can significantly reduce inference cost while maintaining high accuracy. However, most existing routers treat this as a fixed choice among a small set of models, which makes them hard to adapt to new models or changing budget constraints. In this paper, we propose SCOPE (Scalable and Controllable Outcome Performance Estimator), a routing framework that goes beyond model selection by predicting their cost and performance. Trained with reinforcement learning, SCOPE makes reasoning-based predictions by retrieving how models behave on similar problems, rather than relying on fixed model names, enabling it to work with new, unseen models. Moreover, by explicitly predicting how accurate and how expensive a model will be, it turns routing into a dynamic decision problem, allowing users to easily control the trade-off between accuracy and cost. Experiments show that SCOPE is more than just a cost-saving tool. It flexibly adapts to user needs: it can boost accuracy by up to 25.7% when performance is the priority, or cut costs by up to 95.1% when efficiency matters most.
Paper Structure (86 sections, 1 theorem, 39 equations, 18 figures, 4 tables)

This paper contains 86 sections, 1 theorem, 39 equations, 18 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work and Preliminaries
  3. Predictive Routing Methods.
  4. LLM-based Routers.
  5. Motivation: Harmonizing Flexibility and Reasoning.
  6. Roadmap.
  7. Fingerprint Construction
  8. Model Fingerprinting
  9. Retrieval-Augmented Context
  10. Performance Prediction and Model Training
  11. Reasoning Estimator
  12. Data Construction
  13. Scope-60K (Supervision).
  14. Scope-250 (Fingerprinting).
  15. Training Strategy
  16. ...and 71 more sections

Key Result

Proposition 4.1

Under the deterministic tie-breaking rule above, for every $k\in\{1,\ldots,K\}$ and every $\alpha\in(\tau_{k-1},\tau_k)$, the routing decisions satisfy $M_\alpha(x)=M_{\bar{\alpha}_k}(x)$ for all $x\in\mathcal{X}$. Consequently, both $\widehat{C}(\alpha;\mathcal{X})$ and $\widehat{P}(\alpha;\mathcal

Figures (18)

  • Figure 1: Paradigm Comparison. Unlike other LLM-based routers (a) that perform closed-set classification, Scope (b) leverages the model's past behaviors to explicitly predict token length and correctness. This pre-hoc estimation enables generalization to unseen models and facilitates budget-aware decision-making.
  • Figure 2: The Scope Framework. The pipeline consists of three stages: (a) constructing behavioral fingerprints via anchor retrieval; (b) estimating outcome correctness and cost using a reasoning-driven predictor optimized with SFT and GRPO; and (c) selecting the optimal model via a calibrated, budget-aware utility function. Ablation studies of key components are detailed in Section. \ref{['subsec:ablation']}.
  • Figure 3: Composition of Scope-60K. The dataset spans diverse domains, primarily STEM (e.g., Math 29.7%, Chemistry 20.2%, Physics 18.0%, etc) and Humanities (e.g., History 21.4%, Politics 20.7%, Chinese 18.0%, etc).
  • Figure 4: Comparison with Individual Models.Scope shows effective trade-off performance in two distinct regimes. Performance Boosting: With high $\alpha$, Scope transcends the accuracy ceiling of strong baselines (Qwen3-235B and Claude-Sonnet-4.5) by over +24% while maintaining comparable or lower costs. Cost Efficiency: With low $\alpha$, Scope maintains competitive accuracy while slashing inference costs by up to 95.1%. More models' results are provided in Appendix Fig. \ref{['fig:other_comparison']}.
  • Figure 5: Adaptive Model Portfolio.Scope autonomously reconfigures its selection based on $\alpha$. It transitions from low-cost dominance at $\alpha=0$ to a diversified portfolio at $\alpha=1$. More details are provided in Appendix Fig. \ref{['fig:full_portfolio_dist']}.
  • ...and 13 more figures

Theorems & Definitions (2)

  • Proposition 4.1: Correctness of the finite search for the set-level budgeted $\alpha$
  • proof