Multi-Turn Adaptive Prompting Attack on Large Vision-Language Models

TL;DR

This paper introduces MAPA, a two-level, adaptive prompting attack against large vision-language models (LVLMs) to exploit safety gaps by progressively injecting malicious content across turns. At each turn, MAPA selects among text-only, text-plus-vison, and aligned text-plus-vision prompts using a semantic-correlation score $SEM(r,t)$, and across turns it refines the attack trajectory with iterative feedback and a reflection mechanism. Empirical results across HarmBench, JailbreakBench, AdvBench, RedTeam-2K and multiple LVLMs show MAPA achieves 11–35% higher attack success rates than state-of-the-art baselines, including strong performance against GPT-4o-mini with both Default and Advanced Judges. The findings demonstrate the importance of cross-modal coordination and adaptive prompt design for jailbreak efficiency, highlighting the need for robust safety alignments in multimodal systems.

Abstract

Multi-turn jailbreak attacks are effective against text-only large language models (LLMs) by gradually introducing malicious content across turns. When extended to large vision-language models (LVLMs), we find that naively adding visual inputs can cause existing multi-turn jailbreaks to be easily defended. For example, overly malicious visual input will easily trigger the defense mechanism of safety-aligned LVLMs, making the response more conservative. To address this, we propose MAPA: a multi-turn adaptive prompting attack that 1) at each turn, alternates text-vision attack actions to elicit the most malicious response; and 2) across turns, adjusts the attack trajectory through iterative back-and-forth refinement to gradually amplify response maliciousness. This two-level design enables MAPA to consistently outperform state-of-the-art methods, improving attack success rates by 11-35% on recent benchmarks against LLaVA-V1.6-Mistral-7B, Qwen2.5-VL-7B-Instruct, Llama-3.2-Vision-11B-Instruct and GPT-4o-mini.

Figures (14)

• Figure 1: An example of multi-turn dialogues with different input types and a combination of them on Llava-V1.6-Mistral-7B. We apply a state-of-the-art multi-turn jailbreak method for LLMs chainOfAttack to generate text prompts and Stable Diffusion stableDiffusion to produce query-related images. Directly using this method or simply adding images fails to jailbreak the model, whereas carefully selecting less-defendable attack actions across turns progressively elicits more malicious responses. We provide more detailed version of dialogues in Appendix \ref{['sec:fullDialogueExamples']}.
• Figure 2: A visual illustration of MAPA. Right: at each turn, MAPA alterates text-vision attack actions to elicit the most malicious response calculated by semantic correlations. Left: across turns, MAPA adjusts the attack trajectory through iterative back-and-forth refinement, thereby gradually amplifying the maliciousness of responses, improving the jailbreak effectiveness. Reflection: If the current multi-turn attack attempt fails, the subsequent attempt begins by regenerating a chain attack using information from prior failures.
• Figure 3: Distribution of attack actions across turns of MAPA's successful jailbreaks on HarmBench.
• Figure 4: An example of attacking with text-only prompts.
• Figure 5: An example of attacking with text prompts and query-related images.