Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust image classification with multi-modal large language models

Francesco Villani, Igor Maljkovic, Dario Lazzaro, Angelo Sotgiu, Antonio Emanuele Cinà, Fabio Roli

TL;DR

The paper tackles adversarial robustness in image classification by introducing Multi-Shield, a defense that fuses an adversarially trained image classifier with a multimodal vision-language detector (CLIP) to identify semantic misalignment between visual inputs and textual prompts. It defines a rejection mechanism based on agreement between unimodal and multimodal predictions, abstaining when disagreement indicates potential adversarial manipulation. Empirical results across six datasets and multiple models show that Multi-Shield substantially boosts robust accuracy, with average gains of about 32% on CIFAR-10 and 65% on ImageNet, while incurring small reductions in clean accuracy. The work demonstrates that incorporating semantic cross-modal signals provides a practical, scalable augmentation to existing defenses, improving resilience against adaptive attackers without extensive retraining.

Abstract

Deep Neural Networks are vulnerable to adversarial examples, i.e., carefully crafted input samples that can cause models to make incorrect predictions with high confidence. To mitigate these vulnerabilities, adversarial training and detection-based defenses have been proposed to strengthen models in advance. However, most of these approaches focus on a single data modality, overlooking the relationships between visual patterns and textual descriptions of the input. In this paper, we propose a novel defense, MultiShield, designed to combine and complement these defenses with multi-modal information to further enhance their robustness. MultiShield leverages multi-modal large language models to detect adversarial examples and abstain from uncertain classifications when there is no alignment between textual and visual representations of the input. Extensive evaluations on CIFAR-10 and ImageNet datasets, using robust and non-robust image classification models, demonstrate that MultiShield can be easily integrated to detect and reject adversarial examples, outperforming the original defenses.

Robust image classification with multi-modal large language models

TL;DR

The paper tackles adversarial robustness in image classification by introducing Multi-Shield, a defense that fuses an adversarially trained image classifier with a multimodal vision-language detector (CLIP) to identify semantic misalignment between visual inputs and textual prompts. It defines a rejection mechanism based on agreement between unimodal and multimodal predictions, abstaining when disagreement indicates potential adversarial manipulation. Empirical results across six datasets and multiple models show that Multi-Shield substantially boosts robust accuracy, with average gains of about 32% on CIFAR-10 and 65% on ImageNet, while incurring small reductions in clean accuracy. The work demonstrates that incorporating semantic cross-modal signals provides a practical, scalable augmentation to existing defenses, improving resilience against adaptive attackers without extensive retraining.

Abstract

Deep Neural Networks are vulnerable to adversarial examples, i.e., carefully crafted input samples that can cause models to make incorrect predictions with high confidence. To mitigate these vulnerabilities, adversarial training and detection-based defenses have been proposed to strengthen models in advance. However, most of these approaches focus on a single data modality, overlooking the relationships between visual patterns and textual descriptions of the input. In this paper, we propose a novel defense, MultiShield, designed to combine and complement these defenses with multi-modal information to further enhance their robustness. MultiShield leverages multi-modal large language models to detect adversarial examples and abstain from uncertain classifications when there is no alignment between textual and visual representations of the input. Extensive evaluations on CIFAR-10 and ImageNet datasets, using robust and non-robust image classification models, demonstrate that MultiShield can be easily integrated to detect and reject adversarial examples, outperforming the original defenses.

Paper Structure

This paper contains 13 sections, 6 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Adversarial examples
  4. Adversarial Defenses
  5. Multimodal Vision-Language Models
  6. Multi-Shield's Rejection Mechanism
  7. Multishield's Workflow
  8. Problem Formulation
  9. Attacking Multi-Shield
  10. Experiments
  11. Experimental Setup
  12. Experimental Results
  13. Conclusions, Limitations, and Future Works

Figures (2)

  • Figure 1: Illustration of Multi-Shield's operation. An image classifier (1) first processes the input image to generate an initial prediction (2). Simultaneously, the CLIP model (4) performs zero-shot classification using text prompts (3). A rejection score (6) is computed by comparing the agreement between the image classifier's prediction (2) and the CLIP model's output (5). Lastly, Multi-Shield either returns the final prediction or abstains (7), according to the uncertainty in classification. The left part illustrates a clean prediction, while the right part shows Multi-Shield abstaining for an adversarial example.
  • Figure 2: Curves showing the baseline robust accuracy alongside Multi-Shield's robust accuracy and rejection ratio under attacks with varying perturbation sizes ($\varepsilon$) across different robust models on CIFAR-10 (top two plots) and ImageNet (bottom two plots).