Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Models

TL;DR

Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Models introduces IJA, a framework that covertly embeds malicious prompts into images using LSB steganography and couples them with benign-looking image prompts. It formalizes the attack as jointly optimizing the embedded content and textual prompts, aided by adversarial suffixes generated by a surrogate model and an iterative template-optimization loop. Experiments on GPT-4o, Gemini-1.5 Pro, and Qwen2.5-VL-72B achieve over 90% jailbreak success with a mean of about 3 queries and high bypass rates, even under safety defenses. The work analyzes cross-modal safety vulnerabilities and discusses limitations and defenses needed to mitigate such implicit attacks.

Abstract

Multimodal large language models (MLLMs) enable powerful cross-modal reasoning capabilities. However, the expanded input space introduces new attack surfaces. Previous jailbreak attacks often inject malicious instructions from text into less aligned modalities, such as vision. As MLLMs increasingly incorporate cross-modal consistency and alignment mechanisms, such explicit attacks become easier to detect and block. In this work, we propose a novel implicit jailbreak framework termed IJA that stealthily embeds malicious instructions into images via least significant bit steganography and couples them with seemingly benign, image-related textual prompts. To further enhance attack effectiveness across diverse MLLMs, we incorporate adversarial suffixes generated by a surrogate model and introduce a template optimization module that iteratively refines both the prompt and embedding based on model feedback. On commercial models like GPT-4o and Gemini-1.5 Pro, our method achieves attack success rates of over 90% using an average of only 3 queries.

Figures (14)

• Figure 1: Overview of our proposed implicit jailbreak framework for MLLMs. Left: We first optimize an adversarial suffix using a surrogate MLLM, guiding the model to produce a specific malicious output. Right: The optimized prompt is rewritten into an extraction-style template and embedded into an image. If the model fails to decode the malicious instruction, the feedback refines the textual prompt for better attack success.
• Figure 2: Score distribution of the proposed method under MM-SafetyBench dataset.
• Figure 3: Category-wise analysis of GPT-4o on HADES dataset. Our method maintains high ASR and low query cost across categories.
• Figure 4: ASR (%) across three datasets and three MLLMs. Each axis represents a task category.
• Figure 5: Prompt for Red-Teaming LLM.