LLMRouterBench: A Massive Benchmark and Unified Framework for LLM Routing

TL;DR

LLMRouterBench tackles the challenge of evaluating LLM routing methods over large model ensembles by delivering a large-scale, open benchmark and unified framework that covers 21 datasets and 33 models across performance-oriented and performance-cost settings. The study reveals strong model complementarity but shows that many top routing methods achieve similar performance, with several commercial routers failing to outperform a simple Best Single baseline and a persistent gap to Oracle driven mainly by model-recall failures. It also demonstrates that embedding backbones have limited impact and that increasing ensemble size yields diminishing returns, highlighting the value of careful model curation. Furthermore, the framework enables latency-aware analysis, enabling joint optimization across performance, cost, and latency, and provides a path toward more robust, cost-efficient routing in real-world deployments.

Abstract

Large language model (LLM) routing assigns each query to the most suitable model from an ensemble. We introduce LLMRouterBench, a large-scale benchmark and unified framework for LLM routing. It comprises over 400K instances from 21 datasets and 33 models. Moreover, it provides comprehensive metrics for both performance-oriented routing and performance-cost trade-off routing, and integrates 10 representative routing baselines. Using LLMRouterBench, we systematically re-evaluate the field. While confirming strong model complementarity-the central premise of LLM routing-we find that many routing methods exhibit similar performance under unified evaluation, and several recent approaches, including commercial routers, fail to reliably outperform a simple baseline. Meanwhile, a substantial gap remains to the Oracle, driven primarily by persistent model-recall failures. We further show that backbone embedding models have limited impact, that larger ensembles exhibit diminishing returns compared to careful model curation, and that the benchmark also enables latency-aware analysis. All code and data are available at https://github.com/ynulihao/LLMRouterBench.

Figures (10)

• Figure 1: (a) Interest in LLM routing over time, as measured by the cumulative number of related papers and Google Trends. (b) An Overview of LLMRouterBench.
• Figure 2: LLMRouterBench framework integrating Collector, Evaluator, and Adaptor components for standardized evaluation of LLM routing methods.
• Figure 3: Model performance across domains. No single model dominates all domains.
• Figure 4: Performance metrics on LLMRouterBench. $^\dagger$Peak score on the test set.
• Figure 5: Query hardness distribution (by number of correct models) with router accuracy.