Jailbreaks on Vision Language Model via Multimodal Reasoning

TL;DR

Vision-language models exhibit safety fragility due to the modality gap between vision and language. The authors propose a unified, ReAct-driven jailbreak that jointly rewrites prompts and adaptively perturbs images via a Thought–Action–Observation loop, leveraging model feedback to evade safety filters. They introduce a black-box auditing mechanism based on internal reasoning traces and dynamic image filtering strategies to maximize attack success rates. Experiments on VLGuard and SPA-VL with Gemini-2.0-Flash demonstrate higher ASR than baselines, signaling significant safety vulnerabilities and prompting the need for stronger multimodal defenses.

Abstract

Vision-language models (VLMs) have become central to tasks such as visual question answering, image captioning, and text-to-image generation. However, their outputs are highly sensitive to prompt variations, which can reveal vulnerabilities in safety alignment. In this work, we present a jailbreak framework that exploits post-training Chain-of-Thought (CoT) prompting to construct stealthy prompts capable of bypassing safety filters. To further increase attack success rates (ASR), we propose a ReAct-driven adaptive noising mechanism that iteratively perturbs input images based on model feedback. This approach leverages the ReAct paradigm to refine adversarial noise in regions most likely to activate safety defenses, thereby enhancing stealth and evasion. Experimental results demonstrate that the proposed dual-strategy significantly improves ASR while maintaining naturalness in both text and visual domains.

Figures (6)

• Figure 1: Overview of our jailbreak attack method against VLM via iteratively rewriting unsafe prompts and adding noise to unsafe images.
• Figure 2: Examples of ReAct pipeline. Each diagram shows the original input vs. filtered versions across 2 iterations of ReAct Rewriting and Filtering. Left diagram applies ReAct process to image filtering; Right diagram applies ReAct process to prompt rewriting. These transformations reduce harm perceptibility to VLM increasing chance of jailbreaking.
• Figure 3: Examples of adaptive image filtering applied during the jailbreak pipeline. Each set shows the original input (left) and the filtered version (right). Row 1 applies a frequency-domain Discrete Cosine Transform (DCT) filter, Row 2 applies Gaussian blur to smooth visual features, and Row 3 applies image recoloring to shift color distributions. These transformations reduce perceptibility to human viewers while altering features to VLMs.
• Figure 4: More examples to show the process of our proposed bimodal reasoning based jailbreaks. The setup follows Fig. \ref{['fig: ReAct']}.
• Figure 5: Left: Distribution of the number of ReAct iterations required before a prompt was accepted (non-refused). Right: Text Safety Bypass Rate as a function of the number of ReAct attempts. Together, these plots illustrate the efficiency of the adaptive rewriting loop in bypassing refusals.