Learning To See But Forgetting To Follow: Visual Instruction Tuning Makes LLMs More Prone To Jailbreak Attacks
Georgios Pantazopoulos, Amit Parekh, Malvina Nikandrou, Alessandro Suglia
TL;DR
The paper investigates how visual instruction tuning affects the safety of vision-language models by benchmarking three VLMs against their LLM backbones under jailbreaking prompts. It finds that VLMs are more susceptible to jailbreaking, likely due to forgetting safety guardrails during multimodal fine-tuning, and that providing visual context can intensify harmful outputs. The authors propose evaluation benchmarks, data-parity considerations, and safety defenses across all training stages to mitigate these risks and encourage continual safety improvements in multimodal systems. This work highlights the practical significance of safety in multimodal AI, emphasizing that strengthening backbone guardrails alone is insufficient for robust multimodal safety.
Abstract
Augmenting Large Language Models (LLMs) with image-understanding capabilities has resulted in a boom of high-performing Vision-Language models (VLMs). While studying the alignment of LLMs to human values has received widespread attention, the safety of VLMs has not received the same attention. In this paper, we explore the impact of jailbreaking on three state-of-the-art VLMs, each using a distinct modeling approach. By comparing each VLM to their respective LLM backbone, we find that each VLM is more susceptible to jailbreaking. We consider this as an undesirable outcome from visual instruction-tuning, which imposes a forgetting effect on an LLM's safety guardrails. Therefore, we provide recommendations for future work based on evaluation strategies that aim to highlight the weaknesses of a VLM, as well as take safety measures into account during visual instruction tuning.