Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models

TL;DR

Lingua-SafetyBench targets a critical gap in evaluating safety for multilingual vision-language models by disentangling image- and text-dominant risks across ten languages. It presents a large-scale, semantically aligned dataset and a risk-aware translation pipeline that preserves modality-specific semantics, enabling robust cross-lingual safety testing. Experiments on 11 open-source VLLMs reveal opposite risk patterns across modalities and languages, show that scaling yields uneven safety gains, and emphasize that safety alignment cannot be achieved by scaling alone. The work provides open resources and highlights the need for targeted, language- and modality-aware safety strategies to ensure equitable and reliable VLLM deployment.

Abstract

Robust safety of vision-language large models (VLLMs) under joint multilingual and multimodal inputs remains underexplored. Existing benchmarks are typically multilingual but text-only, or multimodal but monolingual. Recent multilingual multimodal red-teaming efforts render harmful prompts into images, yet rely heavily on typography-style visuals and lack semantically grounded image-text pairs, limiting coverage of realistic cross-modal interactions. We introduce Lingua-SafetyBench, a benchmark of 100,440 harmful image-text pairs across 10 languages, explicitly partitioned into image-dominant and text-dominant subsets to disentangle risk sources. Evaluating 11 open-source VLLMs reveals a consistent asymmetry: image-dominant risks yield higher ASR in high-resource languages, while text-dominant risks are more severe in non-high-resource languages. A controlled study on the Qwen series shows that scaling and version upgrades reduce Attack Success Rate (ASR) overall but disproportionately benefit HRLs, widening the gap between HRLs and Non-HRLs under text-dominant risks. This underscores the necessity of language- and modality-aware safety alignment beyond mere scaling.To facilitate reproducibility and future research, we will publicly release our benchmark, model checkpoints, and source code.The code and dataset will be available at https://github.com/zsxr15/Lingua-SafetyBench.Warning: this paper contains examples with unsafe content.

Figures (34)

• Figure 1: Comparison with existing benchmarks. (a) Prior works either evaluate multilingual and multimodal safety separately or lack risk control, often restricted to typography. (b) Lingua-SafetyBench unifies these dimensions with explicit risk attribution, covering diverse languages and visual modalities.
• Figure 2: Pipeline of constructing Lingua-SafetyBench. The pipeline consists of three stages: (1) constructing a multimodal benchmark explicitly partitioned into image- and text-dominant risks; (2) generating multilingual versions via risk-aligned translation; and (3) evaluating model safety using GPT-5.1 and Qwen-Guard to measure attack success rate (ASR).
• Figure 3: Multilingual word clouds of Lingua-SafetyBench. The visualization aggregates keywords from image-dominant risk (including typography and mixed types) and text-dominant risk (i.e., unsafe text queries), highlighting the diverse semantic coverage across ten languages.
• Figure 4: Overall safety performance (ASR) of 11 VLLMs on Lingua-SafetyBench. The evaluation reveals that VLLMs still exhibit significant safety vulnerabilities under multilingual multimodal inputs.
• Figure 5: Safety performance across ten languages. (a) The average ASR of 11 VLLMs reveals significant safety disparities across languages, with English and Norwegian generally safer than others like Finnish or Japanese. (b) A comparison of the Qwen model families (Qwen2-VL, Qwen2.5-VL, and Qwen3-VL) demonstrates that model scaling and iteration consistently improve safety across all tested languages.