SafeVision: Efficient Image Guardrail with Robust Policy Adherence and Explainability

TL;DR

SafeVision tackles the need for efficient, explainable image guardrails that adapt to evolving threats without retraining. It combines a dual-mode VLM-based guardrail with a data-rich VisionHarm framework, a self-refinement training loop, and a custom-weighted loss plus DPO to achieve strong policy adherence and interpretability. VisionHarm-T and VisionHarm-C provide diverse, richly annotated benchmarks enabling rigorous evaluation; SafeVision demonstrates state-of-the-art accuracy and speed, outperforming GPT-4o on VisionHarm-T by 8.6% and VisionHarm-C by 15.5% while achieving over 16x faster inference. The approach supports text-based in-context learning to adapt to new categories, and outputs results in JSON for scalable real-time deployment, marking a substantial step toward policy-aligned, explainable, and practical image guardrails.

Abstract

With the rapid proliferation of digital media, the need for efficient and transparent safeguards against unsafe content is more critical than ever. Traditional image guardrail models, constrained by predefined categories, often misclassify content due to their pure feature-based learning without semantic reasoning. Moreover, these models struggle to adapt to emerging threats, requiring costly retraining for new threats. To address these limitations, we introduce SafeVision, a novel image guardrail that integrates human-like reasoning to enhance adaptability and transparency. Our approach incorporates an effective data collection and generation framework, a policy-following training pipeline, and a customized loss function. We also propose a diverse QA generation and training strategy to enhance learning effectiveness. SafeVision dynamically aligns with evolving safety policies at inference time, eliminating the need for retraining while ensuring precise risk assessments and explanations. Recognizing the limitations of existing unsafe image benchmarks, which either lack granularity or cover limited risks, we introduce VisionHarm, a high-quality dataset comprising two subsets: VisionHarm Third-party (VisionHarm-T) and VisionHarm Comprehensive(VisionHarm-C), spanning diverse harmful categories. Through extensive experiments, we show that SafeVision achieves state-of-the-art performance on different benchmarks. SafeVision outperforms GPT-4o by 8.6% on VisionHarm-T and by 15.5% on VisionHarm-C, while being over 16x faster. SafeVision sets a comprehensive, policy-following, and explainable image guardrail with dynamic adaptation to emerging threats.

Figures (9)

• Figure 1: Overview of the $\textsc{SafeVision}$ image guardrail system. Left:$\textsc{SafeVision}$ operates in dual modes - a rapid $\textsc{classification mode}$ for efficient screening and a $\textsc{comprehension mode}$ that provides both classifications and human-readable explanations. Center:$\textsc{SafeVision}$ follows user-defined safety policies dynamically, eliminating the need for retraining when new threats emerge. Right:$\textsc{SafeVision}$ outputs results directly in JSON format with a lightning-fast inference time of under 100ms per image.
• Figure 2: Overview of the $\textsc{VisionHarm-T}$ creation pipeline. Top: First, a fine-tuned vision classifier performs initial filtering to identify potentially harmful images. Images classified as potentially unsafe (HARM) proceed through the stage of increasingly precise filtering, using a VLM consistency filter, to create a high-density harmful image dataset from a large-scale open-source dataset. Bottom: The VLM QA generator creates question-answer pairs about the image content and policy violations, which are used to construct the $\textsc{VisionHarm-T}$ dataset for training and benchmarking $\textsc{SafeVision}$ and other unsafe image detection models.
• Figure 3: Overview of the $\textsc{SafeVision}$ training pipeline. Left: Model & Policy preparation, including modifications to the tokenizer and the creation of the first version of the guardrail policy. Middle: Self-refinement training, an iterative process involving data cleaning, policy updating, and model fine-tuning to incrementally improve accuracy. Top-right: Post-training, utilizing a custom-weighted loss function to prioritize key tokens and enhance model performance in image guardrail tasks. Bottom-right: Text-based ICL, a text-based in-context learning method that leverages crafted examples to address new harmful categories.
• Figure 4: Top: AUPRC comparison across ten categories in $\textsc{VisionHarm-T}$ shows that $\textsc{SafeVision}$ achieves the highest AUPRC score in all the categories. Middle: The AUPRC scores for baseline VLMs and $\textsc{SafeVision}$ on $\textsc{VisionHarm-C}$. $\textsc{SafeVision}$ achieves the best performance in most categories, andsignificantly outperforming specialized guradrail VLMs. Bottom: The AUPRC scores for baseline VLMs and $\textsc{SafeVision}$ on 8 new categories. $\textsc{SafeVision}$ achieves comparable performance to vanilla VLMs, and significantly outperforming specialized guradrail VLMs.
• Figure 5: Ablation results. (a) The effect of weighted loss ratio on performance. Increasing the weight ratio boosts model performance initially, but excessive ratios lead to performance decline from overfitting.(b) The influence of few-shot example formats on performance. $\textsc{SafeVision}$-8B performs better with detailed, structured examples, while $\textsc{SafeVision}$-2B remains suboptimal across all formats.(c) The impact of the number of few-shot examples on performance. $\textsc{SafeVision}$-2B underperforms, while $\textsc{SafeVision}$-8B's performance improves with more examples, reaches its peak with four and deteriorates with excessive demonstrations.(d) The effectiveness of self-refinement training on performance improvement. $\textsc{SafeVision}$ shows rapid performance gains in the first two epochs; by the fourth epoch, performance stabilizes.