Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedback

TL;DR

This work tackles safety alignment for multimodal large language models by proposing Safe RLHF-V, a dual-preference framework that optimizes for both helpfulness and safety. It introduces BeaverTails-V, the first open-source multimodal safety dataset with dual annotations and multi-level harm labels, along with Beaver-Guard-V, a guardrail system that mitigates unsafe inputs and adversarial probes. The method employs a Lagrangian-based, budget-bound min–max optimization using two dedicated models (RM-V for reward and CM-V for cost) to balance improving helpfulness while restricting harmful outputs. Empirical results show substantial dual gains across multiple models and settings, demonstrating improved safety (+34.2%) and helpfulness (+34.3%) with robust convergence and practical training stability. The work highlights the value of dual supervision and constrained optimization for safer, more capable multimodal AI systems, with plans to extend to additional modalities and evolving safety challenges.

Abstract

Multimodal large language models (MLLMs) are essential for building general-purpose AI assistants; however, they pose increasing safety risks. How can we ensure safety alignment of MLLMs to prevent undesired behaviors? Going further, it is critical to explore how to fine-tune MLLMs to preserve capabilities while meeting safety constraints. Fundamentally, this challenge can be formulated as a min-max optimization problem. However, existing datasets have not yet disentangled single preference signals into explicit safety constraints, hindering systematic investigation in this direction. Moreover, it remains an open question whether such constraints can be effectively incorporated into the optimization process for multi-modal models. In this work, we present the first exploration of the Safe RLHF-V -- the first multimodal safety alignment framework. The framework consists of: $\mathbf{(I)}$ BeaverTails-V, the first open-source dataset featuring dual preference annotations for helpfulness and safety, supplemented with multi-level safety labels (minor, moderate, severe); $\mathbf{(II)}$ Beaver-Guard-V, a multi-level guardrail system to proactively defend against unsafe queries and adversarial attacks. Applying the guard model over five rounds of filtering and regeneration significantly enhances the precursor model's overall safety by an average of 40.9%. $\mathbf{(III)}$ Based on dual preference, we initiate the first exploration of multi-modal safety alignment within a constrained optimization. Experimental results demonstrate that Safe RLHF effectively improves both model helpfulness and safety. Specifically, Safe RLHF-V enhances model safety by 34.2% and helpfulness by 34.3%.

Figures (5)

• Figure 1: Safety annotation distribution of various multimodal safety dataset. We labeled the type of images as safe and unsafe (including minor, moderate, and severe) by expert annotations. We find that existing datasets significantly overestimate the proportion of safe images. In contrast, BeaverTails-V presents a lower proportion of safe images, more accurately reflecting harmful categories.
• Figure 2: Illustration of Beaver-Guard-V’s Moderation Pipeline. (a)&(b): Traditional moderation filters toxic prompts pre-generation, often refusing to respond, which can yield unhelpful or harmful outputs. Beaver-Guard-V adopts a two-stage approach, moderating both prompts and responses post-generation via rejection and regeneration to ensure safe, helpful outputs. (c): It provides fine-grained QA-pair annotations (Safe, Minor, Moderate, Severe) for downstream safety analysis. (d): Across multiple moderation rounds, Beaver-Guard-V significantly lowers attack success rates on models.
• Figure 3: Left: Ablation: static reward shaping and Safe RLHF-V. We find that reward shaping tends to improve single dimension, while our method maintains a consistent improvement for dual objective. w/o and w denote the static reward shaping coefficients and Safe RLHF-V. Right: The trends of $\lambda$, reward, and cost reflect the constrained min-max optimization principle behind the algorithm.
• Figure 4: Pipeline for dataset construction. A three-step process is outlined, involving image collection, prompt generation, and response generation, with annotations for helpfulness and safety. Each step ensures the quality and diversity of generated responses, categorizing them based on harm levels and preferences for helpfulness and safety.
• Figure 5: Example of harmful and unethical themes in BeaverTails-V: A collection of 20 panels categorized into 9 primary harmful domains, each labeled with a number (X-X), where the first X indicates the primary category and the second X represents the subcategory, covering a range of harmful and unethical behaviors.