Defending against adversarial attacks using mixture of experts

TL;DR

This work tackles the challenge of defending ML models against adversarial threats while preserving clean accuracy. It introduces Divided We Fall, a defense that embeds adversarial training into a soft, flat mixture of experts with nine ResNet-18 based experts and a jointly trained gating mechanism, enabling end to end optimization. The approach yields a weighted ensemble decision $y = \sum_{i} w_i(x) f_i(x)$ that balances robustness and accuracy, outperforming state of the art and undefended baselines on CIFAR-10. The results demonstrate that integrating adversarial training within MoE provides practical improvements in adversarial robustness with minimal loss in performance on benign inputs.

Abstract

Machine learning is a powerful tool enabling full automation of a huge number of tasks without explicit programming. Despite recent progress of machine learning in different domains, these models have shown vulnerabilities when they are exposed to adversarial threats. Adversarial threats aim to hinder the machine learning models from satisfying their objectives. They can create adversarial perturbations, which are imperceptible to humans' eyes but have the ability to cause misclassification during inference. Moreover, they can poison the training data to harm the model's performance or they can query the model to steal its sensitive information. In this paper, we propose a defense system, which devises an adversarial training module within mixture-of-experts architecture to enhance its robustness against adversarial threats. In our proposed defense system, we use nine pre-trained experts with ResNet-18 as their backbone. During end-to-end training, the parameters of expert models and gating mechanism are jointly updated allowing further optimization of the experts. Our proposed defense system outperforms state-of-the-art defense systems and plain classifiers, which use a more complex architecture than our model's backbone.

Figures (9)

• Figure 1: Generating an adversarial example on an MNIST digit by adding a carefully crafted perturbation, which remains imperceptible to humans but successfully fools the target models.
• Figure 2: Main components and types of mixture of experts.
• Figure 3: General taxonomy of the threat model.
• Figure 4: General view of the end to end training of our proposed defense system.
• Figure 5: The backbone architecture of DWF network.