VERA-V: Variational Inference Framework for Jailbreaking Vision-Language Models

TL;DR

VERA-V reframes multimodal jailbreak discovery as variational inference over paired text-image prompts, learning a joint posterior to generate diverse, cross-modal adversarial inputs. By integrating typographic rendering, diffusion-based image synthesis, and structured distractors within a latent-prompt and input-transformation pipeline, it achieves state-of-the-art attack success rates while maintaining stealth and transferability across frontier VLMs. The approach combines a LoRA-attacker with a judge-based ELBO optimization via REINFORCE, enabling efficient, scalable red-teaming and adaptive refinement through feedback. Empirical results on HarmBench and HADES demonstrate substantial improvements over prior baselines, including up to 53.75% ASR gains on GPT-4o, and reveal strong cross-model transferability and reduced toxicity detection, highlighting the need for distributional multimodal safety evaluation.

Abstract

Vision-Language Models (VLMs) extend large language models with visual reasoning, but their multimodal design also introduces new, underexplored vulnerabilities. Existing multimodal red-teaming methods largely rely on brittle templates, focus on single-attack settings, and expose only a narrow subset of vulnerabilities. To address these limitations, we introduce VERA-V, a variational inference framework that recasts multimodal jailbreak discovery as learning a joint posterior distribution over paired text-image prompts. This probabilistic view enables the generation of stealthy, coupled adversarial inputs that bypass model guardrails. We train a lightweight attacker to approximate the posterior, allowing efficient sampling of diverse jailbreaks and providing distributional insights into vulnerabilities. VERA-V further integrates three complementary strategies: (i) typography-based text prompts that embed harmful cues, (ii) diffusion-based image synthesis that introduces adversarial signals, and (iii) structured distractors to fragment VLM attention. Experiments on HarmBench and HADES benchmarks show that VERA-V consistently outperforms state-of-the-art baselines on both open-source and frontier VLMs, achieving up to 53.75% higher attack success rate (ASR) over the best baseline on GPT-4o.

Figures (7)

• Figure 1: Single-attack vs. feedback-driven multimodal jailbreaks. CS-DJ CSDJcvpr decomposes harmful queries into typographic sub-images and distractors, producing fixed, one-shot adversarial inputs. In contrast, VERA-V employs an attacker LLM that learns a joint text-image prompt distribution, refines it through optimization with judge feedback, and can subsequently sample diverse adversarial prompts during test time.
• Figure 2: Performance of VERA-V: (a) diversity of generated prompts and (b) scalable attack generation, the flat segment corresponds to the initial optimization phase, after which VERA-V rapidly generates effective and diverse attacks.
• Figure 3: Attacker's System Prompt
• Figure 4: In Context Examples used to sample for attacker system prompt
• Figure 5: Limit Time Budge Experiment