VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety

TL;DR

VLSU tackles the gap in vision-language safety by introducing a principled, scalable framework that separates unimodal signals from joint image-text reasoning. It provides a large, real-image dataset with borderline severity and 17 combinatorial patterns across 15 harm categories, enabling fine-grained diagnostic analysis. Empirical results show that state-of-the-art models excel on unimodal cues but struggle with true cross-modal understanding, with substantial error rates and limited gains from inference-time prompting. The work suggests that real progress requires fundamental advances in multimodal composition and safety alignment, and it offers a benchmark to drive future guardrail development.

Abstract

Safety evaluation of multimodal foundation models often treats vision and language inputs separately, missing risks from joint interpretation where benign content becomes harmful in combination. Existing approaches also fail to distinguish clearly unsafe content from borderline cases, leading to problematic over-blocking or under-refusal of genuinely harmful content. We present Vision Language Safety Understanding (VLSU), a comprehensive framework to systematically evaluate multimodal safety through fine-grained severity classification and combinatorial analysis across 17 distinct safety patterns. Using a multi-stage pipeline with real-world images and human annotation, we construct a large-scale benchmark of 8,187 samples spanning 15 harm categories. Our evaluation of eleven state-of-the-art models reveals systematic joint understanding failures: while models achieve 90%-plus accuracy on clear unimodal safety signals, performance degrades substantially to 20-55% when joint image-text reasoning is required to determine the safety label. Most critically, 34% of errors in joint image-text safety classification occur despite correct classification of the individual modalities, further demonstrating absent compositional reasoning capabilities. Additionally, we find that models struggle to balance refusing unsafe content while still responding to borderline cases that deserve engagement. For example, we find that instruction framing can reduce the over-blocking rate on borderline content from 62.4% to 10.4% in Gemini-1.5, but only at the cost of under-refusing on unsafe content with refusal rate dropping from 90.8% to 53.9%. Overall, our framework exposes weaknesses in joint image-text understanding and alignment gaps in current models, and provides a critical test bed to enable the next milestones in research on robust vision-language safety.

Figures (18)

• Figure 1: Illustrations for the need for joint image-text safety understanding. VLMs either respond helpfully to unsafe queries or over-refuse borderline queries.
• Figure 2: Data generation flow showing image-concept and query generation parameters, image, text and combination generation pipeline and the annotation pipeline using policy.
• Figure 3: Comparison of models on three-class classification accuracy split by severity combinations (safe=S, borderline=B, unsafe=U) in pattern image-text-joint (as defined in Section \ref{['subsection:framework_combinatorics']}). Combinations progress from unimodal-dominated safety signals (left) to those requiring joint vision-language understanding (right). Models struggle as joint understanding becomes critical.
• Figure 4: Correlation of unimodal with combination predictions for sets of all data, subset where the combination prediction is correct and where it is wrong.
• Figure 5: Error breakdown of combination errors by types.