MLLM-as-a-Judge for Image Safety without Human Labeling

TL;DR

The paper tackles zero-shot image safety judgment by querying pre-trained multimodal LLMs against a predefined safety constitution. It introduces CLUE, a multi-stage framework that objectifies rules, filters relevant cases via CLIP, decomposes rules into preconditions, applies debiased token-probability judgments, and falls back to cascaded reasoning when needed. Evaluations on the Objective Safety Bench (OS Bench) show strong performance across multiple MLLMs, outperforming both zero-shot baselines and fine-tuned detectors, with high recall and accuracy. This approach offers a scalable, label-free path for automated content moderation and can serve as an auto-labeler to distill safety judgments into smaller models for practical deployment.

Abstract

Image content safety has become a significant challenge with the rise of visual media on online platforms. Meanwhile, in the age of AI-generated content (AIGC), many image generation models are capable of producing harmful content, such as images containing sexual or violent material. Thus, it becomes crucial to identify such unsafe images based on established safety rules. Pre-trained Multimodal Large Language Models (MLLMs) offer potential in this regard, given their strong pattern recognition abilities. Existing approaches typically fine-tune MLLMs with human-labeled datasets, which however brings a series of drawbacks. First, relying on human annotators to label data following intricate and detailed guidelines is both expensive and labor-intensive. Furthermore, users of safety judgment systems may need to frequently update safety rules, making fine-tuning on human-based annotation more challenging. This raises the research question: Can we detect unsafe images by querying MLLMs in a zero-shot setting using a predefined safety constitution (a set of safety rules)? Our research showed that simply querying pre-trained MLLMs does not yield satisfactory results. This lack of effectiveness stems from factors such as the subjectivity of safety rules, the complexity of lengthy constitutions, and the inherent biases in the models. To address these challenges, we propose a MLLM-based method includes objectifying safety rules, assessing the relevance between rules and images, making quick judgments based on debiased token probabilities with logically complete yet simplified precondition chains for safety rules, and conducting more in-depth reasoning with cascaded chain-of-thought processes if necessary. Experiment results demonstrate that our method is highly effective for zero-shot image safety judgment tasks.

Figures (14)

• Figure 1: Examples showing the challenges for simply querying pre-trained MLLMs for zero-shot image safety judgment.
• Figure 2: Example of the preconditions extracted from the rule.
• Figure 3: Process of calculating token based score. The precondition is considered satisfied if the score is larger than a threshold.
• Figure 4: Approach for mitigating the bias from the non-centric content in the image. We compare the token probability based score (see \ref{['fig:yes_prob']}) of the original image and the image with centric region removed, and consider the image satisfy the precondition if the difference of the score is large enough.
• Figure 5: Process of cascaded reasoning-based judgment.