Do We Really Need Curated Malicious Data for Safety Alignment in Multi-modal Large Language Models?

TL;DR

The paper investigates whether curated malicious data is necessary for safety alignment in multi-modal large language models. It shows that the safety gap between vision-language inputs and text-only prompts largely stems from training data distribution biases, rather than image content or response quality. A practical finetuning approach using benign instruct-following data with explicit rejection sentences significantly improves safety without the need for labor-intensive malicious data collection. The findings suggest that correcting underlying data bias can narrow the safety gap in the vision domain, though limitations remain for image-description tasks and transfer to text-only safety.

Abstract

Multi-modal large language models (MLLMs) have made significant progress, yet their safety alignment remains limited. Typically, current open-source MLLMs rely on the alignment inherited from their language module to avoid harmful generations. However, the lack of safety measures specifically designed for multi-modal inputs creates an alignment gap, leaving MLLMs vulnerable to vision-domain attacks such as typographic manipulation. Current methods utilize a carefully designed safety dataset to enhance model defense capability, while the specific knowledge or patterns acquired from the high-quality dataset remain unclear. Through comparison experiments, we find that the alignment gap primarily arises from data distribution biases, while image content, response quality, or the contrastive behavior of the dataset makes little contribution to boosting multi-modal safety. To further investigate this and identify the key factors in improving MLLM safety, we propose finetuning MLLMs on a small set of benign instruct-following data with responses replaced by simple, clear rejection sentences. Experiments show that, without the need for labor-intensive collection of high-quality malicious data, model safety can still be significantly improved, as long as a specific fraction of rejection data exists in the finetuning set, indicating the security alignment is not lost but rather obscured during multi-modal pretraining or instruction finetuning. Simply correcting the underlying data bias could narrow the safety gap in the vision domain.

Figures (7)

• Figure 1: The main workflow of our study. After demonstrating that the quality of safety finetuning data does not contribute as much as we imagine, we modify the LLaVA-Med li2024llava dataset to create our finetuning dataset. For each data point, we simply pick one round of the conversation and replace their original answer with a clear rejection, without periods and $<$eos$>$ tokens.
• Figure 2: Examples of replies generated under different settings. The reasons are not as high quality as the LLaVA-v1.5. To save space, the image of drugs with the phrase "purchase illegal drugs" is not shown in the box.
• Figure 3: The influence rejection data proportion has on jailbreak defense capability and visual understanding accuracy.
• Figure 4: The comparison of evaluation methods between string matching and Llama-3-Guard-8B. In some subfields, the ASRs from Llama-Guard are lower than string matching.
• Figure 5: Examples of rejection reason generations on MM-SafetyBench. With original visual instruction tuning data, LLaVA-v1.5 could generate high-quality rejection reasons even if the training data do not contain any informative content.