Effective Black-Box Multi-Faceted Attacks Breach Vision Large Language Model Guardrails

TL;DR

This work reveals substantial vulnerabilities in vision-language large models equipped with multi-layered safety defenses. By integrating three complementary attack facets—Visual Attack that injects a toxic system prompt via images, Alignment Breaking Attack that exploits the model's tendency to generate contrasting outputs, and Adversarial Signature that deceives content moderators at the end of responses—the authors demonstrate strong black-box transferability across eight commercial VLLMs, achieving an average attack success rate of 61.56% and surpassing prior methods by over 42 percentage points. The study includes thorough ablations, qualitative analyses, and computational-cost assessments, showing that the attacks can scale to real-world models while revealing gaps in current defenses. The results underscore an urgent need for more robust, holistic defenses and standardized evaluation protocols to mitigate emergent multi-faceted adversarial threats in VLLMs.

Abstract

Vision Large Language Models (VLLMs) integrate visual data processing, expanding their real-world applications, but also increasing the risk of generating unsafe responses. In response, leading companies have implemented Multi-Layered safety defenses, including alignment training, safety system prompts, and content moderation. However, their effectiveness against sophisticated adversarial attacks remains largely unexplored. In this paper, we propose MultiFaceted Attack, a novel attack framework designed to systematically bypass Multi-Layered Defenses in VLLMs. It comprises three complementary attack facets: Visual Attack that exploits the multimodal nature of VLLMs to inject toxic system prompts through images; Alignment Breaking Attack that manipulates the model's alignment mechanism to prioritize the generation of contrasting responses; and Adversarial Signature that deceives content moderators by strategically placing misleading information at the end of the response. Extensive evaluations on eight commercial VLLMs in a black-box setting demonstrate that MultiFaceted Attack achieves a 61.56% attack success rate, surpassing state-of-the-art methods by at least 42.18%.

Figures (14)

• Figure 1: Overview of Multi-Faceted Attack. (a) Multi-Layered Defense strategies employed in VLLMs to enhance safety. (b) Existing attacks (e.g., textual attack gcg and visual jailbreaking image qi2023visual) can breach a single defense layer but fail against multi-layered defenses. (c) Our three attack facets work together break the guardrails and contribute to each others successfully, generating high-quality and genuinely harmful responses.
• Figure 2: Framework of the Multi-Faceted Visual attack. This attack uses gradient-based optimization to create an adversarial image that embeds a harmful prompt, bypassing the safety system prompt and triggering harmful responses.
• Figure 3: Qualitative results of Multi-Faceted Attack with baselines on commercial VLLMs, including GPT-4V (purple), GPT-4o (green), Gemini-2.0-Pro (red), Gemini-1.0-Pro (blue), Mistral-Large (orange), and Llama-3.2-11B-Vision-Instruct. Blue indicates rejection, red denotes harmful responses, and orange represents unrelated responses. The bottom section gives more examples. Further detailed examples are available in the \ref{['sec: multi-facetd_egs']}.
• Figure 4: Comparison of computational costs: (a) Parameters and computations during the attack for Multi-Faceted Attack and Visual-AE. (b) Average success attack time on LlamaGuard.
• Figure 5: A typical failure case of HIMRD attack. Gemini-2.0-Pro responds to the malicious prompt; however the response focuses on giving the guidance, without generating the genuinely harmful tweet. Consequently, a malicious user cannot directly copy and paste the prejudices tweet, but would still need to compose it manually.