Eagle: Efficient Training-Free Router for Multi-LLM Inference
Zesen Zhao, Shuowei Jin, Z. Morley Mao
TL;DR
Eagle tackles scalable, budget-aware routing among multiple LLMs in online environments by fusing global general-ability and local specialized-ability through an ELO-based framework that converts sparse user feedback into full model rankings. It is training-free, leveraging a vector database to identify similar past queries and updating ratings incrementally without retraining. Empirical results on RouterBench show Eagle outperforming strong baselines in AUC and delivering substantial online-adaptation efficiency, with dramatically reduced initialization and update times. The approach enables real-time, low-overhead model selection that maintains high inference quality in dynamic, high-volume LLM serving contexts.
Abstract
The proliferation of Large Language Models (LLMs) with varying capabilities and costs has created a need for efficient model selection in AI systems. LLM routers address this need by dynamically choosing the most suitable model for a given query based on task requirements and budget constraints. However, existing routers face challenges in scalability and real-time adaptation, particularly in high-volume online environments. We present Eagle, a novel LLM routing approach that combines global and local ELO ranking modules to overcome these limitations. By evaluating both general and specialized LLM abilities, Eagle provides a scalable, training-free solution that enhances model selection quality while reducing computational overhead. Our experiments across multiple datasets show Eagle consistently outperforms baseline methods, with improvements of up to 23.52 percent in Area Under Curve (AUC) scores. Moreover, Eagle demonstrates remarkable efficiency, requiring only 1/20 of baseline methods' time for initialization and 100 to 200 times faster incremental updates in online scenarios, making it well-suited for dynamic, high-volume online serving environments.