RouteMoA: Dynamic Routing without Pre-Inference Boosts Efficient Mixture-of-Agents

TL;DR

RouteMoA tackles the cost and latency of traditional mixture-of-agents by introducing a layer-wise dynamic routing framework that pre-filters models with a lightweight, query-aware scorer and then refines candidate quality with a mixture of self- and cross-assessment judges using posterior knowledge from prior outputs. A model-ranking step selects high-potential models by balancing performance, cost, and latency, enabling scalable operation over large and heterogeneous LLM pools. Empirical results show substantial efficiency gains (up to 89.8% cost reduction and 63.6% latency reduction in large pools) while maintaining or improving accuracy across diverse tasks and even strong out-of-distribution generalization. This approach offers a practical and scalable path for efficient multi-LLM collaboration in heterogeneous environments.

Abstract

Mixture-of-Agents (MoA) improves LLM performance through layered collaboration, but its dense topology raises costs and latency. Existing methods employ LLM judges to filter responses, yet still require all models to perform inference before judging, failing to cut costs effectively. They also lack model selection criteria and struggle with large model pools, where full inference is costly and can exceed context limits. To address this, we propose RouteMoA, an efficient mixture-of-agents framework with dynamic routing. It employs a lightweight scorer to perform initial screening by predicting coarse-grained performance from the query, narrowing candidates to a high-potential subset without inference. A mixture of judges then refines these scores through lightweight self- and cross-assessment based on existing model outputs, providing posterior correction without additional inference. Finally, a model ranking mechanism selects models by balancing performance, cost, and latency. RouteMoA outperforms MoA across varying tasks and model pool sizes, reducing cost by 89.8% and latency by 63.6% in the large-scale model pool.

Figures (12)

• Figure 1: Significant variations in model capabilities. Values are normalized to [0,1]. Models exhibit clear specialization: Qwen2.5-Coder leads in coding but lags in biomedical tasks; Qwen2.5-Math excels in mathematics but struggles elsewhere; Bio-Medical-Llama dominates in biomedical knowledge but performs poorly in math and coding; Gemma stands out in reasoning and reading. These distinct profiles make it feasible to predict model performance only based on specific user queries.
• Figure 2: Concept comparison between our RouteMoA and previous MoA-based methods. (a) Classical MoA moa forwards all LLMs in each layer, and concatenates all outputs as the input of the next layer. (b) Sparse MoA smoa introduces an LLM-based judge to select some good responses as the input of the next layer. This reduces the number of input tokens, but still needs to forward all LLMs and another LLM-based judge. (c) RouteMoA uses a lightweight router to select parts of LLMs for inference, significantly reducing computational cost.
• Figure 3: RouteMoA architecture. The framework operates layer-wise (left). At each layer $l$, the router selects a subset of suitable LLMs, whose outputs are aggregated and passed to the next layer. The router (right) consists of two stages: b.1 Mixture of Judges, which includes a scorer (trained as in a. Scorer Training), self-assessment, and cross-assessment. The scorer predicts candidate performance in layer-1 using prior knowledge from the query; subsequent layers refine scores via self- and cross-assessment using posterior knowledge from model outputs. b.2 Model Ranking selects LLMs by balancing performance, cost, and latency.
• Figure 4: Average values of three scorer assessment metrics (Top-1-Hit, Top-3-Hit, and Top-3-Agree) under different training hyperparameters ($\lambda$ and $\alpha$).
• Figure 5: Case study of adjusting wrong scorer predictions with self- and cross-assessment.