Multimodal Safety Is Asymmetric: Cross-Modal Exploits Unlock Black-Box MLLMs Jailbreaks

TL;DR

The work identifies a structural safety weakness in multimodal LLMs: visual alignment can erode the text-based safety guardrails, creating cross-modal jailbreak opportunities. It introduces PolyJailbreak, a reinforcement-learning framework that abstracts vulnerabilities into Atomic Strategy Primitives and jointly optimizes textual and visual prompts to jailbreak various MLLMs. Through extensive evaluation on eight models, PolyJailbreak achieves high attack success and reveals diverse, transferable jailbreak pathways, underscoring the need for modality-aware safety alignment. The study highlights practical risks for real-world deployments and calls for defense strategies that address cross-modal interactions and alignment weaknesses.

Abstract

Multimodal large language models (MLLMs) have demonstrated significant utility across diverse real-world applications. But MLLMs remain vulnerable to jailbreaks, where adversarial inputs can collapse their safety constraints and trigger unethical responses. In this work, we investigate jailbreaks in the text-vision multimodal setting and pioneer the observation that visual alignment imposes uneven safety constraints across modalities in MLLMs, thereby giving rise to multimodal safety asymmetry. We then develop PolyJailbreak, a black-box jailbreak method grounded in reinforcement learning. Initially, we probe the model's attention dynamics and latent representation space, assessing how visual inputs reshape cross-modal information flow and diminish the model's ability to separate harmful from benign inputs, thereby exposing exploitable vulnerabilities. On this basis, we systematize them into generalizable and reusable operational rules that constitute a structured library of Atomic Strategy Primitives, which translate harmful intents into jailbreak inputs through step-wise transformations. Guided by the primitives, PolyJailbreak employs a multi-agent optimization process that automatically adapts inputs against the target models. We conduct comprehensive evaluations on a variety of open-source and closed-source MLLMs, demonstrating that PolyJailbreak outperforms state-of-the-art baselines.

Figures (8)

• Figure 1: Comparison of information flow in LLaVA and Vicuna across dialogue histories (from none to four turns, left to right).
• Figure 2: UMAP visualization of hidden state clustering for benign and malicious instructions (LLaVA vs LLaMA).
• Figure 3: Illustration of the PolyJailbreak workflow, starting with model reconnaissance and vulnerability analysis, followed by attack initialization, and then a reinforcement learning-driven optimization process that iteratively generates and refines jailbreak content until success or the preset step limit is reached.
• Figure 4: Comparison of transfer attack effectiveness.
• Figure 5: Cumulative ASR (%) comparison of PolyJailbreak, RS, and CMA-ES across five optimization steps on target MLLMs.