Toward Universal and Transferable Jailbreak Attacks on Vision-Language Models

TL;DR

The paper tackles the challenge of creating vision-language model jailbreaks that are both universal across targets and transferable across models. It introduces UltraBreak, which couples constrained vision-space optimisation (random transformations and TV regularisation) with semantically weighted targets in the embedding space and a targeted prompt guidance strategy. This combination yields smoother loss landscapes and robust patterns that generalise beyond a single surrogate, outperforming prior gradient-based and typography-based methods across diverse VLMs and benchmarks. The work underscores the amplified attack surface in multimodal models and emphasizes the importance of developing defense strategies that address universal, transferable jailbreak capabilities.

Abstract

Vision-language models (VLMs) extend large language models (LLMs) with vision encoders, enabling text generation conditioned on both images and text. However, this multimodal integration expands the attack surface by exposing the model to image-based jailbreaks crafted to induce harmful responses. Existing gradient-based jailbreak methods transfer poorly, as adversarial patterns overfit to a single white-box surrogate and fail to generalise to black-box models. In this work, we propose Universal and transferable jailbreak (UltraBreak), a framework that constrains adversarial patterns through transformations and regularisation in the vision space, while relaxing textual targets through semantic-based objectives. By defining its loss in the textual embedding space of the target LLM, UltraBreak discovers universal adversarial patterns that generalise across diverse jailbreak objectives. This combination of vision-level regularisation and semantically guided textual supervision mitigates surrogate overfitting and enables strong transferability across both models and attack targets. Extensive experiments show that UltraBreak consistently outperforms prior jailbreak methods. Further analysis reveals why earlier approaches fail to transfer, highlighting that smoothing the loss landscape via semantic objectives is crucial for enabling universal and transferable jailbreaks. The code is publicly available in our \href{https://github.com/kaiyuanCui/UltraBreak}{GitHub repository}.

Figures (6)

• Figure 1: Overview of UltraBreak. UltraBreak introduces two key components to enhance the transferability of optimisation-based jailbreaking images: (1) constraints on the optimisation space and (2) a semantic-driven loss function. The constraints encourage the optimiser to discover robust features that remain invariant across models by incorporating random transformations and a total variation regularisation term. To address the uneven loss landscape introduced by these constraints, the semantic-driven loss aligns optimisation with the target jailbreak semantics rather than individual tokens, yielding more stable and effective training.
• Figure 2: Attack transferability across different surrogate/victim configurations.
• Figure 3: The universal jailbreak patterns obtained with random transformations and TV loss.
• Figure 4: Comparison of loss landscapes: (a) cross-entropy loss and (b–d) semantic loss under different temperature settings $\tau$.
• Figure 5: Comparison of semantic vs. CE loss global landscapes and contours.