Few Tokens Matter: Entropy Guided Attacks on Vision-Language Models

TL;DR

This paper shows that vision-language models (VLMs) exhibit a localized vulnerability in autoregressive decoding: a small subset (~20%) of high-entropy tokens disproportionately governs output trajectories. By concentrating perturbations on these high-entropy decision points, the authors achieve semantic degradation comparable to global attacks but with substantially smaller budgets, and a substantial share of attacked outputs become harmful content. They introduce Entropy-bank Guided Adversarial attacks (EGA), including a transferable variant HiEnt-Bank, which leverages offline token priors to induce strong harm across multiple VLMs and transfer to unseen targets. Across image captioning and VQA tasks, EGA achieves high attack success (≈93–95%) and elevated harmful content rates (≈24–49%), revealing critical safety risks in current VLM safety mechanisms. The work suggests defense strategies should focus on stabilizing high-entropy decision points in autoregressive generation to improve robustness and safety.

Abstract

Vision-language models (VLMs) achieve remarkable performance but remain vulnerable to adversarial attacks. Entropy, a measure of model uncertainty, is strongly correlated with the reliability of VLM. Prior entropy-based attacks maximize uncertainty at all decoding steps, implicitly assuming that every token contributes equally to generation instability. We show instead that a small fraction (about 20%) of high-entropy tokens, i.e., critical decision points in autoregressive generation, disproportionately governs output trajectories. By concentrating adversarial perturbations on these positions, we achieve semantic degradation comparable to global methods while using substantially smaller budgets. More importantly, across multiple representative VLMs, such selective attacks convert 35-49% of benign outputs into harmful ones, exposing a more critical safety risk. Remarkably, these vulnerable high-entropy forks recur across architecturally diverse VLMs, enabling feasible transferability (17-26% harmful rates on unseen targets). Motivated by these findings, we propose Entropy-bank Guided Adversarial attacks (EGA), which achieves competitive attack success rates (93-95%) alongside high harmful conversion, thereby revealing new weaknesses in current VLM safety mechanisms.

Figures (11)

• Figure 1: The examples of high-entropy token manipulation with Qwen2.5-VL-3B, where the red area shows the harmful content.
• Figure 2: The $\Delta$CIDEr distribution w.r.t. the selected top p% high-entropy tokens, showcasing 20% is sufficient.
• Figure 3: Harmful Pie Chart. Nested pies for captioning on three VLMs (left$\to$right: Qwen2.5-VL-7B, InternVL3.5-4B, LLaVA-1.5-7B). The outer ring shows overall outcomes—True (correct & safe), Safe-Wrong (semantic drift but safe), and Harmful (unsafe).The inner ring decomposes Harmful-Wrong into categories: Illegal Activity, Violence, Hate, Self-Harm, Privacy, Sexual Content, and Other.
• Figure 4: Harmful Mass Change, which shows the harmful words of the current high entropy tokens $t$ and their next 10 locations.
• Figure 5: Harmful Rate with different image condition while keeping the textual prefix and target token positions fixed.