Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CSR-Bench: A Benchmark for Evaluating the Cross-modal Safety and Reliability of MLLMs

Yuxuan Liu, Yuntian Shi, Kun Wang, Haoting Shen, Kun Yang

TL;DR

CSR-Bench addresses the gap in evaluating cross-modal safety for multimodal LLMs by introducing a comprehensive benchmark with four stress-testing axes (Safety, Over-rejection, Bias, Hallucination) and 61 fine-grained risk types across 7,405 image-text pairs. It formalizes the distinction between unimodal shortcuts and genuine cross-modal reliability, and adopts a unified protocol with an automated alignment metric (S) and a unimodal baseline for cross-modal comparison. Experiments on 16 MLLMs reveal systematic deficiencies in cross-modal reliability, including vulnerability to cross-modal harm, benign-context over-rejection, social bias from visual cues, and text-driven hallucinations, with an alignment trade-off where safety and bias improvements often increase over-rejection. The findings challenge assumptions that scaling or reasoning augmentation alone improves cross-modal safety, highlighting the need for robust cross-modal alignment for trustworthy, real-world deployment of MLLMs.

Abstract

Multimodal large language models (MLLMs) enable interaction over both text and images, but their safety behavior can be driven by unimodal shortcuts instead of true joint intent understanding. We introduce CSR-Bench, a benchmark for evaluating cross-modal reliability through four stress-testing interaction patterns spanning Safety, Over-rejection, Bias, and Hallucination, covering 61 fine-grained types. Each instance is constructed to require integrated image-text interpretation, and we additionally provide paired text-only controls to diagnose modality-induced behavior shifts. We evaluate 16 state-of-the-art MLLMs and observe systematic cross-modal alignment gaps. Models show weak safety awareness, strong language dominance under interference, and consistent performance degradation from text-only controls to multimodal inputs. We also observe a clear trade-off between reducing over-rejection and maintaining safe, non-discriminatory behavior, suggesting that some apparent safety gains may come from refusal-oriented heuristics rather than robust intent understanding. WARNING: This paper contains unsafe contents.

CSR-Bench: A Benchmark for Evaluating the Cross-modal Safety and Reliability of MLLMs

TL;DR

CSR-Bench addresses the gap in evaluating cross-modal safety for multimodal LLMs by introducing a comprehensive benchmark with four stress-testing axes (Safety, Over-rejection, Bias, Hallucination) and 61 fine-grained risk types across 7,405 image-text pairs. It formalizes the distinction between unimodal shortcuts and genuine cross-modal reliability, and adopts a unified protocol with an automated alignment metric (S) and a unimodal baseline for cross-modal comparison. Experiments on 16 MLLMs reveal systematic deficiencies in cross-modal reliability, including vulnerability to cross-modal harm, benign-context over-rejection, social bias from visual cues, and text-driven hallucinations, with an alignment trade-off where safety and bias improvements often increase over-rejection. The findings challenge assumptions that scaling or reasoning augmentation alone improves cross-modal safety, highlighting the need for robust cross-modal alignment for trustworthy, real-world deployment of MLLMs.

Abstract

Multimodal large language models (MLLMs) enable interaction over both text and images, but their safety behavior can be driven by unimodal shortcuts instead of true joint intent understanding. We introduce CSR-Bench, a benchmark for evaluating cross-modal reliability through four stress-testing interaction patterns spanning Safety, Over-rejection, Bias, and Hallucination, covering 61 fine-grained types. Each instance is constructed to require integrated image-text interpretation, and we additionally provide paired text-only controls to diagnose modality-induced behavior shifts. We evaluate 16 state-of-the-art MLLMs and observe systematic cross-modal alignment gaps. Models show weak safety awareness, strong language dominance under interference, and consistent performance degradation from text-only controls to multimodal inputs. We also observe a clear trade-off between reducing over-rejection and maintaining safe, non-discriminatory behavior, suggesting that some apparent safety gains may come from refusal-oriented heuristics rather than robust intent understanding. WARNING: This paper contains unsafe contents.
Paper Structure (56 sections, 3 equations, 1 figure, 19 tables)

This paper contains 56 sections, 3 equations, 1 figure, 19 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Framework of CSR-Bench
  4. Preliminary
  5. Unimodal Shortcuts.
  6. Cross-modal Reliability.
  7. Safety
  8. Over-rejection: Benign Traps
  9. Bias
  10. Hallucination
  11. Evaluation Metrics
  12. Alignment Accuracy ($S$)
  13. Unimodal Baseline for Comparison
  14. Data Statistics and Quality Assurance
  15. Experiments
  16. ...and 41 more sections

Figures (1)

  • Figure 1: Representative examples of the four evaluation dimensions. Each panel shows the image–query pair and contrasts the expected output with a model-produced bad-case output; Safety, Bias, and Over-rejection use unimodal variants, while Hallucination tests robustness under an intentionally misleading distractor text.