V-Attack: Targeting Disentangled Value Features for Controllable Adversarial Attacks on LVLMs

TL;DR

The paper identifies semantic entanglement in patch token representations as a key obstacle to controllable LVLM attacks and reveals that value features $V$ retain disentangled local semantics. It introduces V-Attack, which combines Self-Value Enhancement and Text-Guided Value Manipulation to precisely relocate source concepts toward target concepts within the value feature space, bypassing entangled patch features. Through extensive experiments on open-source and commercial LVLMs, V-Attack achieves substantially higher attack success rates and demonstrates strong cross-model and cross-task transferability, exposing vulnerabilities in modern visual-language understanding. The work also provides defense-relevant insights, including robustness of attacks to common defenses and considerations for ensuring robust evaluation using LLM-based scoring.

Abstract

Adversarial attacks have evolved from simply disrupting predictions on conventional task-specific models to the more complex goal of manipulating image semantics on Large Vision-Language Models (LVLMs). However, existing methods struggle with controllability and fail to precisely manipulate the semantics of specific concepts in the image. We attribute this limitation to semantic entanglement in the patch-token representations on which adversarial attacks typically operate: global context aggregated by self-attention in the vision encoder dominates individual patch features, making them unreliable handles for precise local semantic manipulation. Our systematic investigation reveals a key insight: value features (V) computed within the transformer attention block serve as much more precise handles for manipulation. We show that V suppresses global-context channels, allowing it to retain high-entropy, disentangled local semantic information. Building on this discovery, we propose V-Attack, a novel method designed for precise local semantic attacks. V-Attack targets the value features and introduces two core components: (1) a Self-Value Enhancement module to refine V's intrinsic semantic richness, and (2) a Text-Guided Value Manipulation module that leverages text prompts to locate source concept and optimize it toward a target concept. By bypassing the entangled patch features, V-Attack achieves highly effective semantic control. Extensive experiments across diverse LVLMs, including LLaVA, InternVL, DeepseekVL and GPT-4o, show that V-Attack improves the attack success rate by an average of 36% over state-of-the-art methods, exposing critical vulnerabilities in modern visual-language understanding. Our code and data are available https://github.com/Summu77/V-Attack.

Figures (22)

• Figure 1: By leveraging rich, disentangled value features instead of commonly used entangled patch features, V-Attack enables precise local semantic manipulations that expose the true vulnerabilities of LVLMs, overcoming the imprecise strategies of existing baselines.
• Figure 2: Illustration of our V-Attack framework. (1) Value features (V) are first extracted from multiple surrogate models. (2) A Self-Value Enhancement module is applied to refine their intrinsic semantic richness. (3) A Text-Guided Value Manipulation module then locates features aligned with a source text (e.g., "dog") and shifts their semantics toward a target text (e.g., "cat"). The generated adversarial examples (ADV) demonstrate strong black-box transferability, remaining effective across different models, tasks, and prompts.
• Figure 3: V-Attack achieves superior local semantic control over baselines. (a) Illustration of the Local Semantic Attack, evaluated using Image Captioning task and VQA task. (b) Comparison on six model-task pairs. V-Attack consistently outperforms baselines, highlighting their limitations in precise local semantic attacks.
• Figure 4: Analysis of CLIP-L/14. (a) Patch tokens attend to regions beyond their own, inducing semantic entanglement; (b) Value features suppress dominant global channels, yielding more uniform, disentangled local features; (c) Value features retain higher, stabler entropy than patch features, indicating richer local semantics; (d) Value features achieve higher peak similarity scores and clear spatial region.
• Figure 5: Adversarial examples generated by V-Attack on GPT-4o in response to the prompt: "Please describe this image." Red text highlights the target concept $t_t$ of the local semantic attack, while green text denotes the non-attacked semantic content.