GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoning

TL;DR

GuardReasoner-VL tackles safety for vision-language models by injecting deliberative reasoning into guard decisions. It builds a large multimodal reasoning corpus GuardReasoner-VLTrain, then cold-starts with Reasoning SFT and subsequently trains online RL with data augmentation, dynamic clipping, and a length-aware reward to balance accuracy and token efficiency. Empirical results across 14 guardrail benchmarks show superior F1 performance, with a token-efficient Eco variant achieving comparable gains at reduced token cost. The work releases datasets, code, and models to spur research on interpretable, generalizable multimodal safety systems.

Abstract

To enhance the safety of VLMs, this paper introduces a novel reasoning-based VLM guard model dubbed GuardReasoner-VL. The core idea is to incentivize the guard model to deliberatively reason before making moderation decisions via online RL. First, we construct GuardReasoner-VLTrain, a reasoning corpus with 123K samples and 631K reasoning steps, spanning text, image, and text-image inputs. Then, based on it, we cold-start our model's reasoning ability via SFT. In addition, we further enhance reasoning regarding moderation through online RL. Concretely, to enhance diversity and difficulty of samples, we conduct rejection sampling followed by data augmentation via the proposed safety-aware data concatenation. Besides, we use a dynamic clipping parameter to encourage exploration in early stages and exploitation in later stages. To balance performance and token efficiency, we design a length-aware safety reward that integrates accuracy, format, and token cost. Extensive experiments demonstrate the superiority of our model. Remarkably, it surpasses the runner-up by 19.27% F1 score on average. We release data, code, and models (3B/7B) of GuardReasoner-VL at https://github.com/yueliu1999/GuardReasoner-VL/

Figures (14)

• Figure 1: Mean Performance of GuardReasoner-VL on Multi-modal Guardrail Benchmarks.
• Figure 2: Overview Training Pipeline of GuardReasoner-VL.It mainly contains three processes, including data curation, model cold-start, and online RL. Concretely, we first build a reasoning corpus, which contains 123K samples with 631K reasoning steps, spanning text, image, and text-image modalities. We cold-start the model via reasoning SFT. Then, we perform data augmentation to improve the difficulty and diversity of the data via safety-aware data concatenation. In addition, we conduct online RL with a dynamic clipping parameter and the designed length-aware safety reward.
• Figure 3: Input Modalities and Distribution of Our Training Data GuardReasoner-VLTrain. It contains 123K samples with 631K reasoning steps, spanning 3 input modalities, including text, image, and text-image. The prompt and response can be classified as harmful or unharmful.
• Figure 4: Safety-Aware Data Concatenation for Data Augmentation. Given two samples with labels $\{\mathcal{X}_1,\mathcal{Y}_1\}$ and $\{\mathcal{X}_2,\mathcal{Y}_2\}$, we generate a new sample $\mathcal{X_\text{new}}$ by concatenating text and merge image. We assign the new label $\mathcal{Y}_{\text{new}}$ as harmful if any of the original labels $\mathcal{Y}_1,\mathcal{Y}_2$ is harmful. It enables the guard model to identity harmful content hidden among predominantly harmless content.
• Figure 5: Ablation Studies of 3B (left) and 7B Models (right) on Prompt Harmfulness Detection. Y-axis denotes F1 score (%), and X-axis denotes model variants in reasoning SFT and online RL.