INFERENCEDYNAMICS: Efficient Routing Across LLMs through Structured Capability and Knowledge Profiling

TL;DR

InferenceDynamics presents a scalable, knowledge- and capability-driven routing framework for selecting among a large pool of specialized LLMs. It introduces RouteMix, a comprehensive benchmark to evaluate generalization with 24 diverse tasks and four out-of-distribution benchmarks. The method jointly scores knowledge and capabilities and performs cost-aware routing, achieving an average improvement of $1.28$ over the best single model while roughly halving the computational budget under constraints. The work demonstrates robust routing in dynamic model pools and across varied domains, with implications for efficient, tailored utilization of the LLM ecosystem and open-source release for reproducibility.

Abstract

Large Language Model (LLM) routing is a pivotal technique for navigating a diverse landscape of LLMs, aiming to select the best-performing LLMs tailored to the domains of user queries, while managing computational resources. However, current routing approaches often face limitations in scalability when dealing with a large pool of specialized LLMs, or in their adaptability to extending model scope and evolving capability domains. To overcome those challenges, we propose InferenceDynamics, a flexible and scalable multi-dimensional routing framework by modeling the capability and knowledge of models. We operate it on our comprehensive dataset RouteMix, and demonstrate its effectiveness and generalizability in group-level routing using modern benchmarks including MMLU-Pro, GPQA, BigGenBench, and LiveBench, showcasing its ability to identify and leverage top-performing models for given tasks, leading to superior outcomes with efficient resource utilization. The broader adoption of Inference Dynamics can empower users to harness the full specialized potential of the LLM ecosystem, and our code will be made publicly available to encourage further research.

Figures (6)

• Figure 1: Quantification of Knowledge and Capability of top 4 models among candidate LLMs.
• Figure 2: LLM performances across 20 datasets in RouteMix. Dataset labels including "PlanBench" indicate subsets of the PlanBench benchmark. For detailed metric information, refer to \ref{['tab:index_set']}.
• Figure 3: Performance Ratio (%) and Cost Ratio (%) variation on GPQA and LiveBench. The "Best Single Model" refers to the most performant LLM for each task.
• Figure 4: Distribution of knowledge domains across 24 datasets in RouteMix. The In-Domain (ID) subset is utilized for quantifying Knowledge and Capability, while the Out-of-Domain (OOD) subset is employed for evaluating the routing algorithm. Dataset labels including "LiveBench" indicate subsets of the LiveBench benchmark, and labels including "NaturalPlan" similarly denote subsets of the NaturalPlan benchmark. The algorithm to compute the normalized proportion is included in \ref{['domain_calculation']}.
• Figure 5: Comparative distribution of router-selected models. Lighter colors signify a higher selection ratio for a given model. The left panel details model selection across evaluation benchmarks using the Optimal Mixed Routing strategy. The right panel illustrates the impact of an increasing cost penalty coefficient ($\beta$) on the model selection distribution.