Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adversarial attacks against Modern Vision-Language Models

Alejandro Paredes La Torre

Abstract

We study adversarial robustness of open-source vision-language model (VLM) agents deployed in a self-contained e-commerce environment built to simulate realistic pre-deployment conditions. We evaluate two agents, LLaVA-v1.5-7B and Qwen2.5-VL-7B, under three gradient-based attacks: the Basic Iterative Method (BIM), Projected Gradient Descent (PGD), and a CLIP-based spectral attack. Against LLaVA, all three attacks achieve substantial attack success rates (52.6%, 53.8%, and 66.9% respectively), demonstrating that simple gradient-based methods pose a practical threat to open-source VLM agents. Qwen2.5-VL proves significantly more robust across all attacks (6.5%, 7.7%, and 15.5%), suggesting meaningful architectural differences in adversarial resilience between open-source VLM families. These findings have direct implications for the security evaluation of VLM agents prior to commercial deployment.

Adversarial attacks against Modern Vision-Language Models

Abstract

We study adversarial robustness of open-source vision-language model (VLM) agents deployed in a self-contained e-commerce environment built to simulate realistic pre-deployment conditions. We evaluate two agents, LLaVA-v1.5-7B and Qwen2.5-VL-7B, under three gradient-based attacks: the Basic Iterative Method (BIM), Projected Gradient Descent (PGD), and a CLIP-based spectral attack. Against LLaVA, all three attacks achieve substantial attack success rates (52.6%, 53.8%, and 66.9% respectively), demonstrating that simple gradient-based methods pose a practical threat to open-source VLM agents. Qwen2.5-VL proves significantly more robust across all attacks (6.5%, 7.7%, and 15.5%), suggesting meaningful architectural differences in adversarial resilience between open-source VLM families. These findings have direct implications for the security evaluation of VLM agents prior to commercial deployment.
Paper Structure (13 sections, 3 equations, 1 figure, 1 table)

This paper contains 13 sections, 3 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Methods
  3. Deployment Environment
  4. Basic Iterative Method
  5. Projected Gradient Descent
  6. CLIP-Based Spectral Attack
  7. Experiments
  8. Experimental Setup
  9. Baseline Evaluation
  10. Attack Success Rate
  11. Differential Robustness Between VLM Families
  12. Limitations
  13. Conclusion

Figures (1)

  • Figure 1: Overview of the adversarial red-teaming pipeline. An adversarially perturbed product image is served through a Flask storefront, captured as a screenshot by a Selenium agent, and passed to the VLM inference server. The VLM returns a structured action that causes the agent to purchase the adversarially targeted product rather than the intended item.