Exposing and Defending the Achilles' Heel of Video Mixture-of-Experts

TL;DR

This work addresses the overlooked vulnerability of video MoE models by dissecting component-level weaknesses in routers and experts. It introduces Temporal Lipschitz-Guided Attacks (TLGA) and its joint variant (J-TLGA) to reveal independent and collaborative weaknesses, and proposes Joint Temporal Lipschitz Adversarial Training (J-TLAT) for hierarchical defense. Experiments on UCF-101 and HMDB-51 across multiple video backbones demonstrate that joint attacks substantially degrade robustness, while J-TLAT achieves meaningful robustness gains with minimal cost. The findings provide a principled framework for analyzing and defending video MoE systems against adversarial perturbations, with implications for safer deployment in safety-critical applications.

Abstract

Mixture-of-Experts (MoE) has demonstrated strong performance in video understanding tasks, yet its adversarial robustness remains underexplored. Existing attack methods often treat MoE as a unified architecture, overlooking the independent and collaborative weaknesses of key components such as routers and expert modules. To fill this gap, we propose Temporal Lipschitz-Guided Attacks (TLGA) to thoroughly investigate component-level vulnerabilities in video MoE models. We first design attacks on the router, revealing its independent weaknesses. Building on this, we introduce Joint Temporal Lipschitz-Guided Attacks (J-TLGA), which collaboratively perturb both routers and experts. This joint attack significantly amplifies adversarial effects and exposes the Achilles' Heel (collaborative weaknesses) of the MoE architecture. Based on these insights, we further propose Joint Temporal Lipschitz Adversarial Training (J-TLAT). J-TLAT performs joint training to further defend against collaborative weaknesses, enhancing component-wise robustness. Our framework is plug-and-play and reduces inference cost by more than 60% compared with dense models. It consistently enhances adversarial robustness across diverse datasets and architectures, effectively mitigating both the independent and collaborative weaknesses of MoE.

Figures (8)

• Figure 1: Conventional attacks vs. ours. Conventional attacks treat the MoE as a whole, overlooking its internal architecture and thus capping the attack strength. Our method performs a component-wise joint attack that explicitly targets the Router and Experts, steering the Router toward weak experts and simultaneously perturbing them. This strategy precisely exposes the Achilles’ Heel (collaborative weaknesses) of MoE while significantly elevating the attack potency.
• Figure 2: Framework of J-TLAT. Temporal Lipschitz Guided Attack for Router (TLGA-R), Temporal Lipschitz Attack for Expert (TLA-E), Temporal Lipschitz Attack for Moe (TLA-M) all are designed for MoE. J-TLAT enhances component-wise and overall robustness hierarchically across three steps per epoch.
• Figure 3: The left figure represents the robustness using different attacks as internal attacks in J-TLAT. The Right figure represents the robustness when using different regularization methods for the external training of J-TLAT on UCF-101.
• Figure 3: Transfer attack performance. The values represent the model's remaining accuracy (%) under different transfer attacks. Lower values indicate a more effective attack. J-TLGA possesses better generalization capabilities.
• Figure 4: Distribution of IoU scores under different attacks on UCF-101. IoU score is a metric used to measure the output consistency of MoE or Router. The greater the deviation from the IoU distribution of clean data, the higher the attack success rate for the adversarial data.