AnyExperts: On-Demand Expert Allocation for Multimodal Language Models with Mixture of Expert

TL;DR

AnyExperts tackles the inefficiency of fixed per-token expert budgets in multimodal Mixture-of-Experts models by introducing a per-token, importance-driven routing framework with a virtual-expert reserve. Tokens are assigned a dynamic total number of expert slots within $[K_{ m min},K_{ m max}]$, with a capped virtual share $\rho_{ m max}$, and a lightweight module predicts token importance $w_i$ to drive allocation and residual state fusion $h_i' = h_i + \alpha w_i h_i$. The method combines importance-aware routing, a balanced load mechanism, and auxiliary losses to stabilize training, achieving improved performance under the same compute budget and maintaining accuracy with fewer real activations across vision, OCR, audio, video, and NLP benchmarks. Empirically, AnyExperts demonstrates strong cross-domain results, robust compute savings (e.g., average real-expert activations reduced from 8 to ~7.2 with no accuracy loss), and informative visualizations of adaptive token importance. This work advances efficient, modality-aware MoE routing with practical implications for scalable, multimodal large models.

Abstract

Multimodal Mixture-of-Experts (MoE) models offer a promising path toward scalable and efficient large vision-language systems. However, existing approaches rely on rigid routing strategies (typically activating a fixed number of experts per token) ignoring the inherent heterogeneity in semantic importance across modalities. This leads to suboptimal compute allocation, where redundant tokens consume as many resources as critical ones. To address this, we propose AnyExperts, a novel on-demand, budget-aware dynamic routing framework that allocates a variable total number of expert slots per token based on its semantic importance. Crucially, to prevent uncontrolled compute growth, the total slots per token are constrained within a fixed range, and each slot is filled by either a real expert or a virtual expert, with the virtual share capped at a small maximum (e.g., 20%). The model then adaptively balances the real-to-virtual ratio per token, assigning more real experts to semantically rich regions and relying more on virtual experts for redundant content. Evaluated across diverse tasks in visual understanding, audio understanding, and NLP understanding, AnyExperts improves performance under the same compute budget. Notably, on general image/video tasks, it achieves comparable accuracy with 40% fewer real expert activations; on text-dense tasks (OCR and NLP), it maintains performance while reducing real expert usage by 10%. These results demonstrate that fine-grained, importance-driven expert allocation significantly enhances both the efficiency and effectiveness of multimodal MoE models.

Figures (10)

• Figure 1: AnyExperts replaces rigid Top-$K$ routing with importance driven expert allocation: tokens from multimodal inputs are assigned a dynamic number of expert slots (composed of real and virtual experts), based on their semantic significance. Informative tokens activate more real experts; redundant ones rely more on virtual computation, ensuring compute is focused precisely where it matters most.
• Figure 2: Overview framework of AnyExperts. For each input token, a lightweight gating network estimates an importance score that dynamically determines the total number of expert. The router then allocates these slots between real experts and virtual expert, enabling per-token adaptive expert activation—i.e., semantically rich tokens activate more real experts, while redundant ones use fewer, achieving fine-grained, budget-aware computation.
• Figure 3: Performance sensitivity to activated experts number under static v.s. dynamic routing. Results for general QA, video QA, OCR and NLP are shown; audio understanding exhibits minimal variation across budgets and is omitted for clarity, with full results provided in the supplementary material.
• Figure 4: Token importance heatmaps for text–image samples. Redder regions indicate higher importance; bluer regions indicate lower importance.
• Figure 5: Token importance analysis for text–video samples. Yellow circles indicate answer targets; bar charts show per-frame aggregated token importance over time.