RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs

TL;DR

The paper investigates routing LLMs, revealing a model-level scaling up phenomenon where a capable router dramatically improves performance as the candidate pool grows, sometimes surpassing the pool's best single model. It introduces RouterEval, a large-scale benchmark built from 12 evaluations and over 200 million performance records across more than 8,500 LLMs to rigorously test router methods. Through systematic experiments, the authors show that existing routers offer limited gains and exhibit biases, underscoring substantial room for improvement while providing a rich open dataset and evaluation framework to accelerate progress. The work highlights practical deployment insights, notably that 3–10 well-chosen candidates can deliver strong performance with favorable cost trade-offs, and it invites the community to expand benchmarks and data resources for routing research.

Abstract

Routing large language models (LLMs) is a new paradigm that uses a router to recommend the best LLM from a pool of candidates for a given input. In this paper, our comprehensive analysis with more than 8,500 LLMs reveals a novel model-level scaling up phenomenon in Routing LLMs, i.e., a capable router can significantly enhance the performance of this paradigm as the number of candidates increases. This improvement can even surpass the performance of the best single model in the pool and many existing strong LLMs, confirming it a highly promising paradigm. However, the lack of comprehensive and open-source benchmarks for Routing LLMs has hindered the development of routers. In this paper, we introduce RouterEval, a benchmark tailored for router research, which includes over 200,000,000 performance records for 12 popular LLM evaluations across various areas such as commonsense reasoning, semantic understanding, etc., based on over 8,500 various LLMs. Using RouterEval, extensive evaluations of existing Routing LLM methods reveal that most still have significant room for improvement. See https://github.com/MilkThink-Lab/RouterEval for all data, code and tutorial.

Figures (7)

• Figure 1: The Overview of Routing LLMs. For each given input, the router distributes it to the appropriate LLM to achieve specific objectives, such as high accuracy, low computational cost, reduced hallucinations, etc. We find that Routing LLMs is a promising paradigm for LLMs to achieve model-level scaling up.
• Figure 2: The Model-level Scaling Up Phenomenon in Routing LLMs. As shown in Section \ref{['sec:poten']}, the Prob. $p$ indicates the performance of the router, with values closer to 1 representing greater similarity to the oracle router's capability. If $p \to 0$, then $r_o(p)$ degenerates into a random sampler. When the router $r_o(p)$ reaches a certain level of capability, it induces a scaling up phenomenon in the Routing LLMs paradigm. Specifically, as the number of LLM candidates increases, performance rapidly improves. "Ref. LLM" denotes a representative LLM with strong performance on given benchmark, such as GPT-4. Further details are provided in Section \ref{['sec:metrics']}. For more examples of model-level scaling up phenomenon, please refer to the Appendix \ref{['sec:more']}.
• Figure 3: The Results on Different Candidate Group.
• Figure 4: The Distribution of Model Parameters. We conduct a statistical analysis of the parameter counts for all LLMs considered in this paper and find that models with 7B parameters are predominant.
• Figure 5: The Distribution of Model Performance Under 12 Benchmarks. We present the performance distribution of the LLMs involved in each of the 12 benchmarks. As shown in Fig. \ref{['fig:stat']}, the majority of LLMs are 7B models, which tend to exhibit relatively weaker performance across the benchmarks from a statistical standpoint.