SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimization

TL;DR

Safety in multimodal LLMs is challenged by cross-modal interactions. The paper proposes SafeGRPO, a self-rewarded safety alignment framework that injects rule-governed rewards into GRPO and leverages SafeTag-VL-3K for verifiable supervision, enabling interpretable, verifiable safety optimization without human labels. It introduces step-guided safety thinking to generate interpretable intermediate safety states and a gated reward that combines tag and behavior signals, expressed as $R_{\text{safety}} = I_{\text{format}} \times (0.5 \times R_{\text{tag}} + 0.5 \times R_{\text{behavior}})$. Empirical results show SafeGRPO improves jailbreaking defense, safety awareness, and reduces over-sensitivity while preserving or even enhancing general multimodal capabilities. This work provides a scalable, verifiable pathway for reasoning-based safety alignment in multimodal systems.

Abstract

Multimodal large language models (MLLMs) have demonstrated impressive reasoning and instruction-following capabilities, yet their expanded modality space introduces new compositional safety risks that emerge from complex text-image interactions. Such cross-modal couplings can produce unsafe semantics even when individual inputs are benign, exposing the fragile safety awareness of current MLLMs. While recent works enhance safety by guiding models to reason about potential risks, unregulated reasoning traces may compromise alignment; although Group Relative Policy Optimization (GRPO) offers self-rewarded refinement without human supervision, it lacks verifiable signals for reasoning safety. To address this, we propose SafeGRPO a self-rewarded multimodal safety alignment framework that integrates rule-governed reward construction into GRPO, enabling interpretable and verifiable optimization of reasoning safety. Built upon the constructed SafeTag-VL-3K dataset with explicit visual, textual, and combined safety tags, SafeGRPO performs step-guided safety thinking to enforce structured reasoning and behavior alignment, substantially improving multimodal safety awareness, compositional robustness, and reasoning stability across diverse benchmarks without sacrificing general capabilities.

Figures (10)

• Figure 1: Limitation of safety alignment. Existing methods rely on large-scale supervised or preference data to improve safety reasoning and enhance multimodal safety awareness.
• Figure 2: Pipeline of SafeGRPO, which aligns multimodal reasoning safety via rule-governed rewards built on the SafeTag-VL-3K dataset.
• Figure 3: Overview of SafeTag-VL-3K. The outer ring illustrates the overall dataset composition, while the table summarizes five tag combinations across modalities, with color bars indicating the ratio of safe and unsafe cases.
• Figure 4: Prompt template for safety-constrained rollouts in SafeGRPO. It guides the model through structured, step-wise reasoning with explicit modality-level tagging and ensures syntactic consistency in generated outputs.
• Figure 5: Case Study comparing SafeGRPO with the base model Qwen3-VL-8B-Thinking on FigStep and SIUO. SafeGRPO accurately identifies unsafe intent and provides clear, well-reasoned refusals. Please refer to \ref{['sec:case study']} for detailed analysis.