Adversarial Attacks of Vision Tasks in the Past 10 Years: A Survey

TL;DR

This paper surveys adversarial attacks in vision over the past decade, contrasting traditional, single-modal attacks with emerging LVLM-era threats. It develops a unified framework around adversariality, transferability, and generalization, and presents detailed threat models, victim models, datasets, and evaluation metrics for both traditional and LVLM contexts. The authors categorize attacks by knowledge, purposes, and techniques, and highlight two LVLM-specific generalizations—Cross-Prompt and Cross-Corpus—along with multimodal attack strategies. They also review defenses across training and inference phases and outline future directions, emphasizing transferability, stealth, physical robustness, and efficiency to guide robust LVLM design and evaluation. Overall, the work offers a comprehensive, actionable synthesis to inform defenses and future exploration in visual adversarial attacks across modalities.

Abstract

With the advent of Large Vision-Language Models (LVLMs), new attack vectors, such as cognitive bias, prompt injection, and jailbreaking, have emerged. Understanding these attacks promotes system robustness improvement and neural networks demystification. However, existing surveys often target attack taxonomy and lack in-depth analysis like 1) unified insights into adversariality, transferability, and generalization; 2) detailed evaluations framework; 3) motivation-driven attack categorizations; and 4) an integrated perspective on both traditional and LVLM attacks. This article addresses these gaps by offering a thorough summary of traditional and LVLM adversarial attacks, emphasizing their connections and distinctions, and providing actionable insights for future research.

Figures (12)

• Figure 1: This figure illustrates key AI security issues and the role of evasion attacks. This paper focuses on evasion attacks, referring to them as adversarial attacks for simplicity. According to elements of security, issues can fall into three categories: resource exhaustion, information extraction, and adversarial attacks. Adversarial attacks can target trainging sets (data poisoning), model parameters/structures (model poisoning), or testing data (evasion attacks). Backdoor attacks are a subset of data poisoning.
• Figure 2: Article Structure. AEs and ATKs denote adversarial examples and attacks respectively. The attack methods in this article are divided into two parts: traditional adversarial attacks (\ref{['sec:PromSetting']}, \ref{['sec:TraditionalAtk']}, \ref{['sec:Motivations']}, and \ref{['sec:Applications']}) and LVLM attacks (\ref{['sec:LVLM']}). Traditional attacks include two phases: a basic strategy phase based on different attack paradigms (\ref{['sec:TraditionalAtk_BasicStrategy']}) and an enhancement phase driven by various motivations (\ref{['sec:TraditionalAtk_AttackEnhancement']}). \ref{['sec:Motivations']} and \ref{['sec:LVLM']} further discuss common motivation types and LVLM-based attacks.
• Figure 3: Taxonomy of Metrics in Adversarial Attack.
• Figure 4: Taxonomies of traditional adversarial attacks. GA, PSO, and QEA refer to Genetic Algorithm, Particle Swarm Optimization, and Quantum-inspired Evolutionary Algorithm, respectively. All three types of algorithms belong to heuristic evolutionary algorithms. Considering the compatibility between strategies, methods from different categories under the same attribute may also be combined to use, such as “iteration-based” and “optimization-based” methods.
• Figure 5: Taxonomy of Motivations for Improving Traditional Adversarial Attacks.