Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships

TL;DR

This paper addresses the vulnerability of vision-language models to multimodal adversarial attacks that perturb both image and text, and the challenge posed by one-to-many image-text relationships. It introduces Multimodal Adversarial Training (MAT) and its augmentation-enhanced variant MAT+ to defend VL models by (i) optimizing with multimodal perturbations and (ii) leveraging 1:N augmentations to better approximate the true multimodal distribution. The authors provide practical optimization strategies for inner and outer loops and establish criteria for effective augmentations, showing that cross-modal, well-aligned, diverse augmentations yield robust improvements. Empirical results across image-text retrieval, visual grounding, and image captioning demonstrate that MAT and especially MAT+ substantially improve robustness against multimodal attacks for CLIP, ALBEF, and BLIP on Flickr30k and COCO datasets, with notable gains over unimodal defenses.

Abstract

Pre-trained vision-language (VL) models are highly vulnerable to adversarial attacks. However, existing defense methods primarily focus on image classification, overlooking two key aspects of VL tasks: multimodal attacks, where both image and text can be perturbed, and the one-to-many relationship of images and texts, where a single image can correspond to multiple textual descriptions and vice versa (1:N and N:1). This work is the first to explore defense strategies against multimodal attacks in VL tasks, whereas prior VL defense methods focus on vision robustness. We propose multimodal adversarial training (MAT), which incorporates adversarial perturbations in both image and text modalities during training, significantly outperforming existing unimodal defenses. Furthermore, we discover that MAT is limited by deterministic one-to-one (1:1) image-text pairs in VL training data. To address this, we conduct a comprehensive study on leveraging one-to-many relationships to enhance robustness, investigating diverse augmentation techniques. Our analysis shows that, for a more effective defense, augmented image-text pairs should be well-aligned, diverse, yet avoid distribution shift -- conditions overlooked by prior research. This work pioneers defense strategies against multimodal attacks, providing insights for building robust VLMs from both optimization and data perspectives. Our code is publicly available at https://github.com/CyberAgentAI/multimodal-adversarial-training.

Figures (10)

• Figure 1: Comparison of adversarial training (AT) methods for robust VL models. (a) Unimodal AT, such as TeCoA mao2022understanding and FARE schlarmann2024robust, robustifies a single modality via unimodal adversarial examples (AEs). However, it overlooks two key aspects of VL tasks: multimodal attacks, where attackers perturb both modalities, and one-to-many cross-modal alignment, where an image has multiple valid descriptions, and vice versa. (b) MAT addresses multimodal attacks by generating multimodal AEs during AT. (c) MAT+ further captures the inherent ambiguity in image-text relationships via one-to-many augmentations.
• Figure 2: The relationships between the three properties of augmentations, (i) alignment, (ii) diversity, and (iii) distribution gap, versus Robust Accuracy against multimodal attack (the overall average of IR@1/TR@1 for CLIP/ALBEF on Flickr/COCO).
• Figure 3: Analysis of the number of augmentations and robustness against SGA attacks for: (a) image augmentations using T-I2I(SD), and (b) text augmentations using I2T(Human).
• Figure 4: Different ways of increasing the number of samples for adversarial training. (a) Original data samples (no increasing), (b) Augmenting orig. data samples without 1:N modeling, (c) Augmenting orig. data samples with 1:N modeling (MAT+), (d) Adding new orig. data samples (oracle).
• Figure 5: Qualitative comparison of image retrieval results under multimodal attack (SGA). We compare the TeCoA-ITR baseline (unimodal defense) with our proposed MAT+ (multimodal defense using I2T(Human) augmentations). Images with a blue border indicate correct retrieval, while those with a red border indicate incorrect retrieval.