SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model

TL;DR

SPA-VL introduces a large-scale, automated safety preference dataset for Vision-Language Models, covering 6 harm domains, 13 categories, and 53 subcategories, with 100,788 four-tuples (question, image, chosen response, rejected response) collected from 12 VLMs. Built to optimize RLHF with PPO and DPO, SPA-VL demonstrates substantial improvements in harmlessness and helpfulness while preserving core capabilities, verified across HarmEval, HelpEval, MM-SafetyBench and AdvBench benchmarks. The authors provide extensive ablations showing benefits of data scale, response diversity, and mixed question types, as well as robustness across backbone models, and they validate alignment quality through human-GPT-annotator consistency checks. The work positions SPA-VL as a key resource for robust, scalable safety alignment for VLMs and outlines future directions toward a unified 3H framework and cross-modal transferability.

Abstract

The emergence of Vision Language Models (VLMs) has brought unprecedented advances in understanding multimodal information. The combination of textual and visual semantics in VLMs is highly complex and diverse, making the safety alignment of these models challenging. Furthermore, due to the limited study on the safety alignment of VLMs, there is a lack of large-scale, high-quality datasets. To address these limitations, we propose a Safety Preference Alignment dataset for Vision Language Models named SPA-VL. In terms of breadth, SPA-VL covers 6 harmfulness domains, 13 categories, and 53 subcategories, and contains 100,788 samples of the quadruple (question, image, chosen response, rejected response). In terms of depth, the responses are collected from 12 open-source (e.g., QwenVL) and closed-source (e.g., Gemini) VLMs to ensure diversity. The construction of preference data is fully automated, and the experimental results indicate that models trained with alignment techniques on the SPA-VL dataset exhibit substantial improvements in harmlessness and helpfulness while maintaining core capabilities. SPA-VL, as a large-scale, high-quality, and diverse dataset, represents a significant milestone in ensuring that VLMs achieve both harmlessness and helpfulness.

Figures (12)

• Figure 1: Overview of SPA-VL Dataset Construction. It is built in three stages: 1) Image Collection, 2) Questions Constrution and 3) Preference Construction. The dataset examples shows vision-question-preferences pairs that comprise three types of questions: easy questions, hard questions, and hard statements.
• Figure 2: Case study comparing responses from the original model, the model trained with DPO and PPO on our SPA-VL.
• Figure 3: Impact of Data Scale on Alignment Model Performance. Line plots illustrate the effect of varying data quantities ($100$, $1k$, $5k$, $10k$, $30k$, and $90k$) on the performance metrics of alignment models for both PPO and DPO methods. (a) Shows the Harm Score (%) on EvalHarm (b) Shows the Average Attack Success Rate (ASR %) on MM-SafetyBench (c) Shows ASR (%) on vanilla and suffix in AdvBench (d) Shows the Help Score (%) on Anthropic-Helpful.
• Figure 4: Presentation of our dataset across six primary domains and fifteen secondary categories and 53 Tertiary categories.
• Figure 5: Examples from MM-SafetyBench