Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

XSPA: Crafting Imperceptible X-Shaped Sparse Adversarial Perturbations for Transferable Attacks on VLMs

Chengyin Hu, Jiaju Han, Xuemeng Sun, Qike Zhang, Yiwei Wei, Ang Li, Chunlei Meng, Xiang Chen, Jiahuan Long

Abstract

Vision-language models (VLMs) rely on a shared visual-textual representation space to perform tasks such as zero-shot classification, image captioning, and visual question answering (VQA). While this shared space enables strong cross-task generalization, it may also introduce a common vulnerability: small visual perturbations can propagate through the shared embedding space and cause correlated semantic failures across tasks. This risk is particularly important in interactive and decision-support settings, yet it remains unclear whether VLMs are robust to highly constrained, sparse, and geometrically fixed perturbations. To address this question, we propose X-shaped Sparse Pixel Attack (XSPA), an imperceptible structured attack that restricts perturbations to two intersecting diagonal lines. Compared with dense perturbations or flexible localized patches, XSPA operates under a much stricter attack budget and thus provides a more stringent test of VLM robustness. Within this sparse support, XSPA jointly optimizes a classification objective, cross-task semantic guidance, and regularization on perturbation magnitude and along-line smoothness, inducing transferable misclassification as well as semantic drift in captioning and VQA while preserving visual subtlety. Under the default setting, XSPA modifies only about 1.76% of image pixels. Experiments on the COCO dataset show that XSPA consistently degrades performance across all three tasks. Zero-shot accuracy drops by 52.33 points on OpenAI CLIP ViT-L/14 and 67.00 points on OpenCLIP ViT-B/16, while GPT-4-evaluated caption consistency decreases by up to 58.60 points and VQA correctness by up to 44.38 points. These results suggest that even highly sparse and visually subtle perturbations with fixed geometric priors can substantially disrupt cross-task semantics in VLMs, revealing a notable robustness gap in current multimodal systems.

XSPA: Crafting Imperceptible X-Shaped Sparse Adversarial Perturbations for Transferable Attacks on VLMs

Abstract

Vision-language models (VLMs) rely on a shared visual-textual representation space to perform tasks such as zero-shot classification, image captioning, and visual question answering (VQA). While this shared space enables strong cross-task generalization, it may also introduce a common vulnerability: small visual perturbations can propagate through the shared embedding space and cause correlated semantic failures across tasks. This risk is particularly important in interactive and decision-support settings, yet it remains unclear whether VLMs are robust to highly constrained, sparse, and geometrically fixed perturbations. To address this question, we propose X-shaped Sparse Pixel Attack (XSPA), an imperceptible structured attack that restricts perturbations to two intersecting diagonal lines. Compared with dense perturbations or flexible localized patches, XSPA operates under a much stricter attack budget and thus provides a more stringent test of VLM robustness. Within this sparse support, XSPA jointly optimizes a classification objective, cross-task semantic guidance, and regularization on perturbation magnitude and along-line smoothness, inducing transferable misclassification as well as semantic drift in captioning and VQA while preserving visual subtlety. Under the default setting, XSPA modifies only about 1.76% of image pixels. Experiments on the COCO dataset show that XSPA consistently degrades performance across all three tasks. Zero-shot accuracy drops by 52.33 points on OpenAI CLIP ViT-L/14 and 67.00 points on OpenCLIP ViT-B/16, while GPT-4-evaluated caption consistency decreases by up to 58.60 points and VQA correctness by up to 44.38 points. These results suggest that even highly sparse and visually subtle perturbations with fixed geometric priors can substantially disrupt cross-task semantics in VLMs, revealing a notable robustness gap in current multimodal systems.

Paper Structure

This paper contains 16 sections, 21 equations, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Sparse and Structured Attacks on Vision Models
  4. Attacks and Robustness Studies on Vision-Language Models
  5. Automatic Evaluation for Open-Ended Tasks
  6. Method
  7. Problem Definition
  8. X-shaped Sparse Structural Mask
  9. Cross-task Joint Objective
  10. Mask-constrained Optimization
  11. Experiments
  12. Experimental Setup
  13. Main Results
  14. Grad-CAM-Based Attention Heatmap Analysis
  15. Ablation Studies
  16. ...and 1 more sections

Figures (5)

  • Figure 1: Qualitative zero-shot classification examples. The top row shows clean images, and the bottom row shows XSPA adversarial images. Despite the fixed X-shaped sparse perturbation, clear prediction shifts are observed.
  • Figure 2: Qualitative examples on image captioning and visual question answering. Compared with clean outputs, XSPA adversarial outputs exhibit clear semantic drift in captions and more error-prone answers in VQA, leading to lower GPT-4-based scores.
  • Figure 3: Grad-CAM attention heatmaps for clean and adversarial samples. Clean examples focus more strongly on the main object, while adversarial ones show more diffuse or shifted responses.
  • Figure 4: Iteration ablation on four CLIP backbones. The attack success rate increases steadily with more optimization steps and saturates near 200 iterations across different encoder architectures, showing a stable convergence trend under diverse encoder designs.
  • Figure 5: Effect of perturbation budget on attack performance and perturbation statistics. Larger budgets improve attack success rate and increase perturbation-related metrics.