Steering MoE LLMs via Expert (De)Activation

TL;DR

SteerMoE treats the MoE router as an interpretable control signal by identifying behavior-linked experts through paired prompts and selectively adjusting router scores at inference. The method detects experts whose activation patterns correlate with distinct behaviors using risk differences, then biases routing to activate or deactivate them via a soft steering mechanism that preserves the multi-expert mixture. Empirically, steering toward safety and faithfulness improves performance on RAG grounding and safety benchmarks (up to +27% faithfulness and +20% safety), while unsafe steering can substantially degrade safety and even bypass guardrails when combined with jailbreak prompts (-100%). The work also reveals vulnerabilities in MoE routing, showing that a sparse subnetwork of experts governs alignment, and highlights the need for robust, router-aware safety strategies alongside interpretability benefits in MoE LLMs.

Abstract

Mixture-of-Experts (MoE) in Large Language Models (LLMs) routes each token through a subset of specialized Feed-Forward Networks (FFN), known as experts. We present SteerMoE, a framework to steer MoE models by detecting and controlling behavior-associated experts. We detect key experts by comparing how often they activate between paired inputs that demonstrate opposite behaviors (e.g., safe vs. unsafe). By selectively activating or deactivating such experts during inference, we control behaviors like faithfulness and safety without fine-tuning. Across 11 benchmarks and 6 LLMs, our steering raises safety by up to +20% and faithfulness by +27%. Alternatively, unsafe steering drops safety by -41% alone, and -100% when combined with existing jailbreak methods, bypassing all safety guardrails. Overall, SteerMoE offers a lightweight, effective, and widely applicable test-time control, while revealing unique vulnerabilities in MoE LLMs. https://github.com/adobe-research/SteerMoE

Figures (10)

• Figure 1: Steering MoE models by routing through behavior-linked experts at inference enables lightweight, interpretable control. Red and green FFNs are controlled by our method; others follow the router's choice. Generations are from Qwen3-30B-A3B. (See more examples in Table \ref{['tab:examples_main']})
• Figure 2: Comparison of steered versus non-steered model performance on faithfulness benchmarks. Accuracy is the proportion of examples in which the response remains faithful to the content of the provided document. The MC Test benchmark serves as a control dataset to ensure that the model's general QA performance remains stable after steering. Modifying expert routing during inference improves performance on faithfulness benchmarks. (More models in Fig. \ref{['fig:steering_faithfulness_all']})
• Figure 3: Comparison of steered versus non-steered model performance on safety benchmarks shows that MoE steering via token routing can significantly adjust safety levels in both directions. Safety is measured by Llama-Guard-3-8B on harmful prompts, with "Fluency" as a control to verify fluency. (More models in Fig. \ref{['fig:steering_safety_full']})
• Figure A.1: Pairplot of different scoring methods (Risk Difference, Log-Odds Ratio, and Paired t-test) for the detection of faithfulness related experts.
• Figure A.2: The effect of the number of manipulated experts on the fluency of Qwen3. Deactivating experts has a softer effect than activating.