SGHA-Attack: Semantic-Guided Hierarchical Alignment for Transferable Targeted Attacks on Vision-Language Models

TL;DR

SGHA-Attack addresses the vulnerability of vision-language models to transfer-based targeted attacks in black-box settings. It introduces Semantic-Guided Anchor Injection to form a diverse anchor pool and hierarchical alignment modules HVSA and CLSS to enforce target semantics across intermediate visual layers and cross-modal representations. The method yields stronger targeted transferability across open-source and commercial VLMs and remains robust under preprocessing defenses, while maintaining visual fidelity. This multi-granularity, cross-modal supervision highlights the importance of internal feature hierarchies for robust semantic control and informs potential defenses against semantic hijacking.

Abstract

Large vision-language models (VLMs) are vulnerable to transfer-based adversarial perturbations, enabling attackers to optimize on surrogate models and manipulate black-box VLM outputs. Prior targeted transfer attacks often overfit surrogate-specific embedding space by relying on a single reference and emphasizing final-layer alignment, which underutilizes intermediate semantics and degrades transfer across heterogeneous VLMs. To address this, we propose SGHA-Attack, a Semantic-Guided Hierarchical Alignment framework that adopts multiple target references and enforces intermediate-layer consistency. Concretely, we generate a visually grounded reference pool by sampling a frozen text-to-image model conditioned on the target prompt, and then carefully select the Top-K most semantically relevant anchors under the surrogate to form a weighted mixture for stable optimization guidance. Building on these anchors, SGHA-Attack injects target semantics throughout the feature hierarchy by aligning intermediate visual representations at both global and spatial granularities across multiple depths, and by synchronizing intermediate visual and textual features in a shared latent subspace to provide early cross-modal supervision before the final projection. Extensive experiments on open-source and commercial black-box VLMs show that SGHA-Attack achieves stronger targeted transferability than prior methods and remains robust under preprocessing and purification defenses.

Figures (8)

• Figure 1: Illustration of the proposed SGHA-Attack framework targeting diverse VLMs.
• Figure 2: Overview of the proposed SGHA-Attack framework for generating transferable targeted adversarial examples. The framework consists of three key components: SGAI, HVSA, and CLSS.
• Figure 3: Visual comparison of adversarial examples generated by various attack methods.
• Figure 4: Qualitative results of our attack on multiple open-source VLMs. The target text prompt is shown above each image pair. For each example, the left image is the clean original with its caption, and the right image is the adversarial example generated by our method ($\epsilon=8/255$) with the model-generated caption. The adversarial captions closely align with the target texts, while the images remain visually similar to the originals.
• Figure 5: Visualization of targeted adversarial attacks on commercial black-box VLMs. We display the adversarial image, the target text, and the actual response generated by the model.