What Gets Activated: Uncovering Domain and Driver Experts in MoE Language Models

TL;DR

This work tackles interpretability gaps in Mixture-of-Experts (MoE) language models by introducing domain experts and driver experts as distinct functional components. It defines entropy-based domain specialization metrics and causal-effect measures to identify activated experts and quantify their influence on predictions, using $H_i(D_j)$, $A_i(D_j)$, $S_i(D_j)$, and $CE(e_i)=D_{KL}\bigl(P(X)\Vert Q_i(X)\bigr)$. Across three MoE LLMs and three domains (SA, MMLU, Math), the study finds a small subset of experts with domain specialization or strong causal impact, while general experts dominate most activations. Tokens occurring early in sentences are more likely to trigger driver experts, and experimentally increasing the weights of domain or driver experts yields consistent performance gains, illustrating the practical value of task-aware routing. Overall, the results enhance MoE interpretability by revealing activation patterns, token–expert associations, and the potential for targeted routing to improve accuracy and reasoning tasks.

Abstract

Most interpretability work focuses on layer- or neuron-level mechanisms in Transformers, leaving expert-level behavior in MoE LLMs underexplored. Motivated by functional specialization in the human brain, we analyze expert activation by distinguishing domain and driver experts. In this work, we study expert activation in MoE models across three public domains and address two key questions: (1) which experts are activated, and whether certain expert types exhibit consistent activation patterns; and (2) how tokens are associated with and trigger the activation of specific experts. To answer these questions, we introduce entropy-based and causal-effect metrics to assess whether an expert is strongly favored for a particular domain, and how strongly expert activation contributes causally to the model's output, thus identify domain and driver experts, respectively. Furthermore, we explore how individual tokens are associated with the activation of specific experts. Our analysis reveals that (1) Among the activated experts, some show clear domain preferences, while others exert strong causal influence on model performance, underscoring their decisive roles. (2) tokens occurring earlier in a sentence are more likely to trigger the driver experts, and (3) adjusting the weights of domain and driver experts leads to significant performance gains across all three models and domains. These findings shed light on the internal mechanisms of MoE models and enhance their interpretability.

Figures (10)

• Figure 1: Expert-level Certainty-Weighted Activation Score (CWAS) for DeepSeek-MoE (left), Qwen-MoE (middle), and Mixtral-8×7B (right).
• Figure 2: Comparison of expert activation behavior across tasks: (a) activation rates of each expert type, and (b) the distribution of activated experts within each task.
• Figure 3: Layer-wise causal rates.
• Figure 3: Tokens associated with domain and driver experts.
• Figure 4: Proportional causal influence of tokens by sentence position under the DeepSeek model. The horizontal bar charts illustrate token contributions across sentence positions for three domains (SA, MMLU, and Math).