When Routing Collapses: On the Degenerate Convergence of LLM Routers

TL;DR

The paper identifies routing collapse in LLM routing, where increasing user budgets cause routers to over-rely on the strongest model, wasting compute and undermining cost savings. It shows this is driven by a mismatch between predicting scalar scores and making discrete, rank-based decisions, especially when model performance margins are small. To address this, it introduces EquiRouter, a ranking-focused router that uses model-conditioned representations and a ranking-loss objective to directly supervise per-query model orderings, promoting fair consideration of cheaper models. Empirically, EquiRouter achieves substantial cost reductions (≈17% on RouterBench and ≈12% on MMR-Bench) while maintaining or improving performance, and it introduces the Routing Collapse Index (RCI) to quantify and monitor collapse across benchmarks and even under out-of-domain conditions.

Abstract

LLM routing aims to achieve a favorable quality--cost trade-off by dynamically assigning easy queries to smaller models and harder queries to stronger ones. However, across both unimodal and multimodal settings, we uncover a pervasive yet underexplored failure mode in existing routers: as the user's cost budget increases, routers systematically default to the most capable and most expensive model even when cheaper models already suffice. As a result, current routers under-utilize small models, wasting computation and monetary cost and undermining the core promise of routing; we term this phenomenon routing collapse. We attribute routing collapse to an objective--decision mismatch: many routers are trained to predict scalar performance scores, whereas routing decisions ultimately depend on discrete comparisons among candidate models. Consequently, small prediction errors can flip relative orderings and trigger suboptimal selections. To bridge this gap, we propose EquiRouter, a decision-aware router that directly learns model rankings, restoring the role of smaller models and mitigating routing collapse. On RouterBench, EquiRouter reduces cost by about 17\% at GPT-4-level performance compared to the strongest prior router. Our code is available at https://github.com/AIGNLAI/EquiRouter.

Figures (11)

• Figure 1: Visualization of routing collapse on RouterBench hu2024routerbench: under existing routers, as the cost budget increases, the call rate of the strongest model (GPT-4) rapidly saturates near 100%, indicating that routers nearly always select the largest model and largely ignore cheaper alternatives.
• Figure 2: Left: An illustration of routing collapse: existing routers over-select the strongest model even when smaller models are sufficient. Right: Per-model call frequency of the Oracle router on RouterBench, where lighter colors indicate lower-cost models and darker colors indicate higher-cost models.
• Figure 3: Left: Routing collapse observed on MMR-Bench. Middle: Feasible-set margin distribution $\Delta(q;C)$ on RouterBench under a large budget, showing that small margins (near-ties) are prevalent. Right: Impact of additive Gaussian noise on Oracle per-model performance labels.
• Figure 4: Overview of EquiRouter: given a query and a set of candidate models, the router constructs query-conditioned model representations and predicts their per-query ranking with a decision-aligned, ranking-aware objective.
• Figure 5: Average cost required to achieve varying performance targets on RouterBench. Lower curves indicate more efficient use of models and better cost-performance trade-offs.