The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models

TL;DR

The paper investigates whether Mixture-of-Experts (MoE) models truly specialize by domain or converge to a domain-invariant core. It introduces COMMITTEEAUDIT, a post-hoc framework that analyzes routing at the level of expert groups, identifying Standing Committees via cross-domain invariants and Pareto-optimality across three MoE models and the MMLU benchmark. The findings reveal a compact, domain-invariant Standing Committee that absorbs the majority of routing mass, anchoring reasoning and syntax while peripheral experts handle domain-specific knowledge; this indicates a strong centralization bias in sparse routing and suggests that load-balancing objectives may counter natural optimization. These insights have practical implications for designing function-aware routing and core-periphery MoE architectures, potentially improving efficiency and interpretability while highlighting new avenues for robust, targeted model analysis.

Abstract

Mixture of Experts models are widely assumed to achieve domain specialization through sparse routing. In this work, we question this assumption by introducing COMMITTEEAUDIT, a post hoc framework that analyzes routing behavior at the level of expert groups rather than individual experts. Across three representative models and the MMLU benchmark, we uncover a domain-invariant Standing Committee. This is a compact coalition of routed experts that consistently captures the majority of routing mass across domains, layers, and routing budgets, even when architectures already include shared experts. Qualitative analysis further shows that Standing Committees anchor reasoning structure and syntax, while peripheral experts handle domain-specific knowledge. These findings reveal a strong structural bias toward centralized computation, suggesting that specialization in Mixture of Experts models is far less pervasive than commonly believed. This inherent bias also indicates that current training objectives, such as load-balancing losses that enforce uniform expert utilization, may be working against the model's natural optimization path, thereby limiting training efficiency and performance.

Figures (34)

• Figure 1: From domain-specific intuition to empirically observed expert sharing in Mixture-of-Experts models. (a) The Intuition: The ideal "Divide-and-Conquer" strategy assumes disjoint sets of experts for different domains. (b) The Observation: Empirical routing patterns reveal a Standing Committee (e.g., Experts E4 and E5) that is consistently activated across disparate domains (Math, Legal, Biology), acting as a generalist core hidden within the routed experts.
• Figure 2: Overview of the COMMITTEEAUDIT framework.
• Figure 3: Evidence of standing committees in MoE models. (a) Layer-wise concentration of top-k experts across tasks. For each model, the solid line shows the mean normalized weight assigned to the top-$k$ experts at each layer, and the shaded region denotes one standard deviation. High and stable values indicate a small subset of experts ("standing committees") consistently absorbs most routing mass. (b–d) Lorenz curves reveal that only a small subset of experts accounts for most contributions (other Lorenz curves are shown in Appendix \ref{['Con_con_ana']}), showing that these committees are highly centralized rather than uniformly shared.
• Figure 4: Cross-layer stability of routed experts across models, measured by Jaccard similarity between top-$k$ expert sets over domains. All three MoE models maintain high overlap ($\geq 0.8$ for most layers), showing that the same experts are repeatedly selected despite changes in input domain and network depth.
• Figure 5: Dynamics of standing committees in OLMoE under different routing budgets. We show all 16 layers. (a) Relative contribution of committee members remains high and does not vanish when $k$ increases. (b) The size of the identified committees stays small and changes only mildly with depth and $k$, indicating a compact but persistent core of experts.