SAFEx: Analyzing Vulnerabilities of MoE-Based LLMs via Stable Safety-critical Expert Identification

TL;DR

SAFEx identifies a distinct MoE-specific safety risk, termed positional vulnerability, where safety-aligned behaviors depend on a small subset of routing experts. It introduces a Stability-based Expert Selection workflow to robustly identify safety-critical experts and then categorizes them into Harmful Content Detection (HCDG) and Harmful Response Control (HRCG) groups. Through linear probing and expert masking experiments across several MoE LLMs, including Qwen3-30B-A3B, SAFEx demonstrates that perturbing or masking a small fraction of experts substantially weakens model safety under harmful prompts, without full-model retraining. The work additionally shows that lightweight LoRA-based weight merging targeting these experts can improve safety alignment under jailbreak conditions, offering a compute-efficient, deployment-friendly defense. Overall, SAFEx provides a framework for MoE-aware safety analysis and targeted expert-level interventions that could guide future robustness improvements in routed architectures.

Abstract

Large language models with Mixture-of-Experts (MoE) architectures achieve efficiency and scalability, yet their routing mechanisms introduce safety alignment challenges insufficiently addressed by techniques developed for dense models. In this work, the MoE-specific safety risk of positional vulnerability-that safety-aligned behaviors rely on specific expert modules-is formalized and systematically analyzed. An analytical framework, SAFEx, is presented to robustly identify, characterize, and validate safety-critical experts via a stability-based expert selection procedure, and to decompose them into two functional groups: the Harmful Content Detection Group (HCDG), which specializes in identifying and recognizing harmful content within user inputs, and the Harmful Response Control Group (HRCG), which specializes in controlling and enforcing model behaviors to generate appropriate safety responses. Expert-level interventions are conducted to probe causality and to test mitigation. Targeted masking of SAFEx-selected experts reveals that safety behavior is highly concentrated. On Qwen3-30B-A3B, configured with 48 MoE-FFN layers and 128 experts per layer under top-8 routing (48x128=6,144 experts in total), disabling 12 selected experts reduces the refusal rate by 22%. In addition, lightweight adaptation is performed using LoRA under three configurations-the HRCG, the union of HCDG and HRCG, and all experts-and the resulting updates are composed through negative weight merging targeted at the HRCG, leading to improved refusal under adversarial prompts without full-model retraining. These results establish positional vulnerability as a distinct MoE-specific safety challenge and provide a practical, compute-efficient pathway for expert-level safety interventions within routed architectures.

Figures (8)

• Figure 1: Positional vulnerability of MoE architecture in current LLMs. (a) Normal harmful request is successfully rejected by MoE. (b) Harmful request passed by MoE due to the masking attack. (c) Harmful request passed by MoE due to the jailbreak attack. (d) The proposed framework SAFEx enables analysis of expert activation patterns and functional roles.
• Figure 2: Overall workflow of SAFEx.
• Figure 3: Activation probability visualization of E_top(D_Regular), E_top(D_Jailbreak), and E_top(D_Benign) for three MoE models. The horizontal dashed lines represent the theoretical mean activation probability $k/K$ under each MoE configuration.
• Figure 4: Performance comparison of linear probes trained on safety-relevant experts from $\mathcal{E}_{\text{id}}$ versus randomly selected experts. Box plots illustrate accuracy, precision, recall, and F1-score metrics across different model architectures. Results consistently show superior performance of linear probes based on experts identified within $\mathcal{E}_{\text{id}}$, indicating these experts encode specialized features related to safety-sensitive content identification.
• Figure 5: Expert-level weight merging pipeline.