Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers

Yang Li

TL;DR

This work challenges the prevailing reliance on complex learned routers for LLM routing by showing that well-tuned kNN-based routing can closely match or outperform sophisticated models across text and multi-modal tasks. It introduces standardized routing benchmarks and a first vision-language routing dataset to enable fair comparisons, and provides both empirical results and theoretical guarantees that locality in embedding space yields strong, low-sample-complexity routing signals. The findings suggest a practical shift toward simple, interpretable baselines that can reduce deployment overhead while maintaining performance. Overall, the paper reframes LLM routing as a locality-exploiting, non-parametric problem with broad implications for efficiency and accessibility in multi-model AI ecosystems.

Abstract

As large language models (LLMs) grow in scale and specialization, routing--selecting the best model for a given input--has become essential for efficient and effective deployment. While recent methods rely on complex learned routing strategies, their dependence on disparate training data and evaluation setups makes comparison and generalization difficult. In this work, we revisit LLM routing through the lens of simplicity. We show that a well-tuned k-Nearest Neighbors (kNN) approach not only matches but often outperforms state-of-the-art learned routers across diverse tasks. To support systematic evaluation, we introduce a suite of standardized routing benchmarks spanning instruction-following, question-answering, and reasoning tasks, as well as the first multi-modal routing dataset involving visual inputs. Our findings reveal that the locality properties of model performance in embedding space enable simple non-parametric methods to achieve strong routing decisions with lower sample complexity than parametric approaches. This challenges the prevailing trend toward sophisticated architectures and highlights the importance of thoroughly evaluating simple baselines before investing in complex solutions. To support reproducibility and further exploration, we will release all benchmarks and code upon publication.

Rethinking Predictive Modeling for LLM Routing: When Simple kNN Beats Complex Learned Routers

TL;DR

This work challenges the prevailing reliance on complex learned routers for LLM routing by showing that well-tuned kNN-based routing can closely match or outperform sophisticated models across text and multi-modal tasks. It introduces standardized routing benchmarks and a first vision-language routing dataset to enable fair comparisons, and provides both empirical results and theoretical guarantees that locality in embedding space yields strong, low-sample-complexity routing signals. The findings suggest a practical shift toward simple, interpretable baselines that can reduce deployment overhead while maintaining performance. Overall, the paper reframes LLM routing as a locality-exploiting, non-parametric problem with broad implications for efficiency and accessibility in multi-model AI ecosystems.

Abstract

As large language models (LLMs) grow in scale and specialization, routing--selecting the best model for a given input--has become essential for efficient and effective deployment. While recent methods rely on complex learned routing strategies, their dependence on disparate training data and evaluation setups makes comparison and generalization difficult. In this work, we revisit LLM routing through the lens of simplicity. We show that a well-tuned k-Nearest Neighbors (kNN) approach not only matches but often outperforms state-of-the-art learned routers across diverse tasks. To support systematic evaluation, we introduce a suite of standardized routing benchmarks spanning instruction-following, question-answering, and reasoning tasks, as well as the first multi-modal routing dataset involving visual inputs. Our findings reveal that the locality properties of model performance in embedding space enable simple non-parametric methods to achieve strong routing decisions with lower sample complexity than parametric approaches. This challenges the prevailing trend toward sophisticated architectures and highlights the importance of thoroughly evaluating simple baselines before investing in complex solutions. To support reproducibility and further exploration, we will release all benchmarks and code upon publication.
Paper Structure (45 sections, 2 theorems, 33 equations, 3 figures, 12 tables)

This paper contains 45 sections, 2 theorems, 33 equations, 3 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Background and Related Works
  3. Predictive LLM Routing
  4. Routing Benchmarks
  5. Text-Based Routing Benchmarks
  6. Vision-Language Routing Benchmarks
  7. Evaluation Protocol
  8. Routing Approaches
  9. Quantitative Results
  10. Theoretical Analysis
  11. Local Structure of Model Performance
  12. Sample Complexity Advantage
  13. Discussion and Conclusion
  14. Additional Related Works
  15. LLM Inference Optimization
  16. ...and 30 more sections

Key Result

Theorem 1

For a query distribution $\mathcal{D}$ with $\delta$-locality in utility space: (a) A kNN router requires a training sample size of $\Theta\left(\frac{C_{\mathcal{X}, d}}{\delta^d} \cdot \log\left(\frac{1}{\alpha}\right)\right)$ to achieve expected regret $O(\epsilon(\delta))$ with probability $1-\a

Figures (3)

  • Figure 1: As the embedding distance between prompt pairs increases, the agreement between their model performance scores decreases, demonstrating the locality property in the prompt-performance space.
  • Figure D.1: Cost-Quality tradeoff for text-based routing benchmarks using model selection approaches.
  • Figure D.2: Cost-Quality tradeoff for VLM routing benchmarks using model selection approaches.

Theorems & Definitions (4)

  • Definition 1: $\delta$-Locality
  • Theorem 1
  • Theorem 1
  • proof