BadPatches: Routing-aware Backdoor Attacks on Vision Mixture of Experts

TL;DR

This paper investigates backdoor vulnerabilities in vision Mixture of Experts (MoE) models that use patch-based processing. It introduces BadPatches, a routing-aware trigger design that ensures the complete trigger is observed by every patch-based expert, achieving high attack success rates even at extremely low poisoning rates (e.g., $0.01\%$) on pMoE and across MoE-based vision transformers like $V$-MoE and $E^3$. Across datasets, BadPatches generally outperforms routing-agnostic triggers, demonstrating strong ASR with minimal poisoning while preserving benign accuracy; the authors also analyze how patch routing and hyperparameters influence attack effectiveness. They further evaluate a defense based on fine-pruning, finding that pruning alone is insufficient, but combining pruning with fine-tuning significantly reduces ASR (e.g., routing-agnostic: $96.6\%\rightarrow1.9\%$, BadPatches: $100\%\rightarrow\approx23\%$). The work highlights the practical threat to MoE vision systems and suggests extending routing-aware backdoor concepts to text-based MoE architectures in future work.

Abstract

Mixture of Experts (MoE) architectures have gained popularity for reducing computational costs in deep neural networks by activating only a subset of parameters during inference. While this efficiency makes MoE attractive for vision tasks, the patch-based processing in vision models introduces new methods for adversaries to perform backdoor attacks. In this work, we investigate the vulnerability of vision MoE models for image classification, specifically the patch-based MoE (pMoE) models and MoE-based vision transformers, against backdoor attacks. We propose a novel routing-aware trigger application method BadPatches, which is designed for patch-based processing in vision MoE models. BadPatches applies triggers on image patches rather than on the entire image. We show that BadPatches achieves high attack success rates (ASRs) with lower poisoning rates than routing-agnostic triggers and is successful at poisoning rates as low as 0.01% with an ASR above 80% on pMoE. Moreover, BadPatches is still effective when an adversary does not have complete knowledge of the patch routing configuration of the considered models. Next, we explore how trigger design affects pMoE patch routing. Finally, we investigate fine-pruning as a defense. Results show that only the fine-tuning stage of fine-pruning removes the backdoor from the model.

Figures (6)

• Figure 1: Examples of a clean CIFAR-10 dog image and how the application of different triggers affects the image. The top row shows routing-agnostic triggers. The bottom row shows BadPatches triggers.
• Figure 2: Patch routings for a CIFAR-10 dog image in models backdoored with routing-agnostic triggers. Every row shows which patches each of the four experts processes for the corresponding image.
• Figure 3: Patch routings for a CIFAR-10 dog image in models backdoored with BadPatches triggers. Every row shows which patches each of the four experts processes for the corresponding image.
• Figure 4: CIFAR-10 image of a dog poisoned with the routing-agnostic square trigger and the corresponding patch routings before and after fine-pruning. The top row shows the patch routing before fine-pruning, and the bottom row shows the patch routing after fine-pruning.
• Figure 5: Patch routing for each expert of a clean pMoE model.