LightRouter: Towards Efficient LLM Collaboration with Minimal Overhead

TL;DR

LightRouter introduces a token-efficient, two-stage routing framework for orchestrating multiple open-source LLMs. By probing all candidate models with brief boot tokens, selectively filtering top performers, and aggregating only the best outputs, it achieves strong accuracy while substantially reducing inference costs. The approach is validated across diverse knowledge and reasoning benchmarks, outperforming single models and competitive ensembles, and it relies entirely on open-source models. The work offers practical insights into adaptive scoring, dynamic specialization, and robust multi-model coordination with broad potential impact for cost-conscious AI deployments.

Abstract

The rapid advancement of large language models has unlocked remarkable capabilities across a diverse array of natural language processing tasks. However, the considerable differences among available LLMs-in terms of cost, performance, and computational demands-pose significant challenges for users aiming to identify the most suitable model for specific tasks. In this work, we present LightRouter, a novel framework designed to systematically select and integrate a small subset of LLMs from a larger pool, with the objective of jointly optimizing both task performance and cost efficiency. LightRouter leverages an adaptive selection mechanism to identify models that require only a minimal number of boot tokens, thereby reducing costs, and further employs an effective integration strategy to combine their outputs. Extensive experiments across multiple benchmarks demonstrate that LightRouter matches or outperforms widely-used ensemble baselines, achieving up to a 25% improvement in accuracy. Compared with leading high-performing models, LightRouter achieves comparable performance while reducing inference costs by up to 27%. Importantly, our framework operates without any prior knowledge of individual models and relies exclusively on inexpensive, lightweight models. This work introduces a practical approach for efficient LLM selection and provides valuable insights into optimal strategies for model combination.

Figures (7)

• Figure 1: Illustration of the LightRouter architecture. The framework routes prompts through a selector agent and multiple layers of aggregator agents, progressively combining outputs from a pool of candidate models to generate the final response. Different agent roles are color-coded.
• Figure 2: Grouped bar chart of MT-Bench scores for all evaluated models and methods. Each group represents a model, with bars indicating the average, 1st turn, and 2nd turn scores. LightRouter is highlighted in red and with a bold outline for emphasis.
• Figure 3: Boxplot comparison of individual candidate scores and aggregated MoA score across six benchmarks. Outliers are shown as small circles. The red dot denotes the MoA score for each task.
• Figure 4: Performance distribution of different LLMs across repeated runs on MT-Bench. Our method (LightRouter) shows both high average performance and low variance.
• Figure 5: Performance of different LLMs when used as aggregators.