Vulnerability-Aware Robust Multimodal Adversarial Training

TL;DR

Vulnerability-Aware Robust Multimodal Adversarial Training (VARMAT) addresses modulus-specific robustness gaps in multimodal models by estimating per-modality vulnerability through a first-order approximation and applying a gradient-norm based regularization in feature space. It introduces vulnerability weights with a temperature-scaled Softmax and a targeted regularization term to balance vulnerabilities across modalities, compatible with fast adversarial training. Empirical results on CMU-MOSEI, UR-FUNNY, and AVMNIST show robust gains up to 12.73%, 22.21%, and 11.19% respectively, revealing a critical blind spot in prior multimodal defenses. The approach preserves accuracy while improving efficiency, demonstrating the practical impact of vulnerability-aware, modality-differentiated optimization for multimodal robustness.

Abstract

Multimodal learning has shown significant superiority on various tasks by integrating multiple modalities. However, the interdependencies among modalities increase the susceptibility of multimodal models to adversarial attacks. Existing methods mainly focus on attacks on specific modalities or indiscriminately attack all modalities. In this paper, we find that these approaches ignore the differences between modalities in their contribution to final robustness, resulting in suboptimal robustness performance. To bridge this gap, we introduce Vulnerability-Aware Robust Multimodal Adversarial Training (VARMAT), a probe-in-training adversarial training method that improves multimodal robustness by identifying the vulnerability of each modality. To be specific, VARMAT first explicitly quantifies the vulnerability of each modality, grounded in a first-order approximation of the attack objective (Probe). Then, we propose a targeted regularization term that penalizes modalities with high vulnerability, guiding robust learning while maintaining task accuracy (Training). We demonstrate the enhanced robustness of our method across multiple multimodal datasets involving diverse modalities. Finally, we achieve {12.73%, 22.21%, 11.19%} robustness improvement on three multimodal datasets, revealing a significant blind spot in multimodal adversarial training.

Figures (5)

• Figure 1: Adversarial robustness under varying attack strengths $\lambda \in [0,0.5]$ for different modalities, showing significant differences in modality-specific vulnerabilities.
• Figure 2: Comparison of vulnerability-aware attack and VARMAT with previous indiscriminate adversarial training.
• Figure 3: Comparison of robustness between different methods in different temperatures.
• Figure 4: Comparison of computational time between fast adversarial training methods.
• Figure 5: Comparison of single-modality PGD attack performance across fast adversarial training methods ($\lambda=0.5$).