Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing the "Immunity" of Mixture-of-Experts Networks for Adversarial Defense

Qiao Han, yong huang, xinling Guo, Yiteng Zhai, Yu Qin, Yao Yang

TL;DR

Immunity presents a robust adversarial defense by augmenting Mixture-of-Experts with Random Switch Gates and Grad-CAM-guided regularizers. It introduces two novel losses, a heatmap-based mutual information objective and a position-stability regularizer, to diversify and stabilize expert representations, while enabling random gating during inference to hinder targeted attacks. Through extensive CIFAR-10/100 experiments, Immunity demonstrates improved resilience against FGSM, BIM, MIM, and PGD attacks, particularly under adversarial training, and provides interpretable heatmaps that reveal specialized expert focus. The work highlights the benefits of ensemble diversity, causality-focused learning, and heatmap-based regularization for practical, robust AI systems.

Abstract

Recent studies have revealed the vulnerability of Deep Neural Networks (DNNs) to adversarial examples, which can easily fool DNNs into making incorrect predictions. To mitigate this deficiency, we propose a novel adversarial defense method called "Immunity" (Innovative MoE with MUtual information \& positioN stabilITY) based on a modified Mixture-of-Experts (MoE) architecture in this work. The key enhancements to the standard MoE are two-fold: 1) integrating of Random Switch Gates (RSGs) to obtain diverse network structures via random permutation of RSG parameters at evaluation time, despite of RSGs being determined after one-time training; 2) devising innovative Mutual Information (MI)-based and Position Stability-based loss functions by capitalizing on Grad-CAM's explanatory power to increase the diversity and the causality of expert networks. Notably, our MI-based loss operates directly on the heatmaps, thereby inducing subtler negative impacts on the classification performance when compared to other losses of the same type, theoretically. Extensive evaluation validates the efficacy of the proposed approach in improving adversarial robustness against a wide range of attacks.

Enhancing the "Immunity" of Mixture-of-Experts Networks for Adversarial Defense

TL;DR

Immunity presents a robust adversarial defense by augmenting Mixture-of-Experts with Random Switch Gates and Grad-CAM-guided regularizers. It introduces two novel losses, a heatmap-based mutual information objective and a position-stability regularizer, to diversify and stabilize expert representations, while enabling random gating during inference to hinder targeted attacks. Through extensive CIFAR-10/100 experiments, Immunity demonstrates improved resilience against FGSM, BIM, MIM, and PGD attacks, particularly under adversarial training, and provides interpretable heatmaps that reveal specialized expert focus. The work highlights the benefits of ensemble diversity, causality-focused learning, and heatmap-based regularization for practical, robust AI systems.

Abstract

Recent studies have revealed the vulnerability of Deep Neural Networks (DNNs) to adversarial examples, which can easily fool DNNs into making incorrect predictions. To mitigate this deficiency, we propose a novel adversarial defense method called "Immunity" (Innovative MoE with MUtual information \& positioN stabilITY) based on a modified Mixture-of-Experts (MoE) architecture in this work. The key enhancements to the standard MoE are two-fold: 1) integrating of Random Switch Gates (RSGs) to obtain diverse network structures via random permutation of RSG parameters at evaluation time, despite of RSGs being determined after one-time training; 2) devising innovative Mutual Information (MI)-based and Position Stability-based loss functions by capitalizing on Grad-CAM's explanatory power to increase the diversity and the causality of expert networks. Notably, our MI-based loss operates directly on the heatmaps, thereby inducing subtler negative impacts on the classification performance when compared to other losses of the same type, theoretically. Extensive evaluation validates the efficacy of the proposed approach in improving adversarial robustness against a wide range of attacks.
Paper Structure (24 sections, 14 equations, 2 figures, 3 tables)

This paper contains 24 sections, 14 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Adversarial Attacks and Defences
  4. CAM-based Explanations for Deep Learning Interpretability
  5. Mutual Information Aided Mixture of Experts for Independent Representations
  6. Background and Motivation
  7. Methodology
  8. Enhancing Mixture of Experts Robustness via Novel Regularizations
  9. Interpreting Experts through Grad-CAM
  10. Simplified MI for Expert Diversity
  11. Regularizing Experts via Position Stability Index
  12. Objective and Training Techniques
  13. Experiment
  14. Experiment Setting
  15. Dataset
  16. ...and 9 more sections

Figures (2)

  • Figure 1: The Architecture of The Proposed “ Immunity”
  • Figure 2: Visualization of “ Immunity” Expert Heatmaps and Original Input Images