Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Doubao 1.8, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5

Xingjun Ma, Yixu Wang, Hengyuan Xu, Yutao Wu, Yifan Ding, Yunhan Zhao, Zilong Wang, Jiabin Hua, Ming Wen, Jianan Liu, Ranjie Duan, Yifeng Gao, Yingshui Tan, Yunhao Chen, Hui Xue, Xin Wang, Wei Cheng, Jingjing Chen, Zuxuan Wu, Bo Li, Yu-Gang Jiang

TL;DR

This paper presents an integrated safety evaluation of seven frontier frontier-models across language, vision–language, and image generation using a unified protocol that combines benchmark tests, adversarial jailbreaks, multilingual evaluation across 18 languages, and regulatory-compliance checks. It reveals a highly heterogeneous safety landscape, with GPT-5.2 generally leading across modalities and other models showing trade-offs between benchmark safety, adversarial robustness, multilingual generalization, and governance compliance. Adversarial prompts expose brittleness in both language and vision–language domains, and image generation safety shows that even strong refusals do not guarantee risk-free outputs under sophisticated attacks. The findings underscore the multidimensional nature of safety in frontier models and argue for standardized, multi-modal evaluation frameworks to guide responsible development and deployment.

Abstract

The rapid evolution of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) has produced substantial gains in reasoning, perception, and generative capability across language and vision. However, whether these advances yield commensurate improvements in safety remains unclear, in part due to fragmented evaluation practices limited to single modalities or threat models. In this report, we present an integrated safety evaluation of 7 frontier models: GPT-5.2, Gemini 3 Pro, Qwen3-VL, Doubao 1.8, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5. We evaluate each model across language, vision-language, and image generation settings using a unified protocol that integrates benchmark evaluation, adversarial evaluation, multilingual evaluation, and compliance evaluation. Aggregating our evaluations into safety leaderboards and model safety profiles across multiple evaluation modes reveals a sharply heterogeneous safety landscape. While GPT-5.2 demonstrates consistently strong and balanced safety performance across evaluations, other models exhibit pronounced trade-offs among benchmark safety, adversarial alignment, multilingual generalization, and regulatory compliance. Both language and vision-language modalities show significant vulnerability under adversarial evaluation, with all models degrading substantially despite strong results on standard benchmarks. Text-to-image models achieve relatively stronger alignment in regulated visual risk categories, yet remain brittle under adversarial or semantically ambiguous prompts. Overall, these results show that safety in frontier models is inherently multidimensional--shaped by modality, language, and evaluation scheme, underscoring the need for standardized safety evaluations to accurately assess real-world risk and guide responsible model development and deployment.

A Safety Report on GPT-5.2, Gemini 3 Pro, Qwen3-VL, Doubao 1.8, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5

TL;DR

This paper presents an integrated safety evaluation of seven frontier frontier-models across language, vision–language, and image generation using a unified protocol that combines benchmark tests, adversarial jailbreaks, multilingual evaluation across 18 languages, and regulatory-compliance checks. It reveals a highly heterogeneous safety landscape, with GPT-5.2 generally leading across modalities and other models showing trade-offs between benchmark safety, adversarial robustness, multilingual generalization, and governance compliance. Adversarial prompts expose brittleness in both language and vision–language domains, and image generation safety shows that even strong refusals do not guarantee risk-free outputs under sophisticated attacks. The findings underscore the multidimensional nature of safety in frontier models and argue for standardized, multi-modal evaluation frameworks to guide responsible development and deployment.

Abstract

The rapid evolution of Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) has produced substantial gains in reasoning, perception, and generative capability across language and vision. However, whether these advances yield commensurate improvements in safety remains unclear, in part due to fragmented evaluation practices limited to single modalities or threat models. In this report, we present an integrated safety evaluation of 7 frontier models: GPT-5.2, Gemini 3 Pro, Qwen3-VL, Doubao 1.8, Grok 4.1 Fast, Nano Banana Pro, and Seedream 4.5. We evaluate each model across language, vision-language, and image generation settings using a unified protocol that integrates benchmark evaluation, adversarial evaluation, multilingual evaluation, and compliance evaluation. Aggregating our evaluations into safety leaderboards and model safety profiles across multiple evaluation modes reveals a sharply heterogeneous safety landscape. While GPT-5.2 demonstrates consistently strong and balanced safety performance across evaluations, other models exhibit pronounced trade-offs among benchmark safety, adversarial alignment, multilingual generalization, and regulatory compliance. Both language and vision-language modalities show significant vulnerability under adversarial evaluation, with all models degrading substantially despite strong results on standard benchmarks. Text-to-image models achieve relatively stronger alignment in regulated visual risk categories, yet remain brittle under adversarial or semantically ambiguous prompts. Overall, these results show that safety in frontier models is inherently multidimensional--shaped by modality, language, and evaluation scheme, underscoring the need for standardized safety evaluations to accurately assess real-world risk and guide responsible model development and deployment.
Paper Structure (56 sections, 23 figures, 12 tables)

This paper contains 56 sections, 23 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Purpose of This Report
  3. Evaluation Protocol
  4. Language, Vision–Language, and Image Generation Safety.
  5. Multilingual Evaluation.
  6. Benchmark and Adversarial Evaluations.
  7. Diversity Over Exhaustiveness.
  8. Summary of Results
  9. Safety Leaderboard
  10. Safety Profiling
  11. Language Safety
  12. Benchmark Evaluation
  13. Experimental Setup
  14. Evaluation Results
  15. Example Responses
  16. ...and 41 more sections

Figures (23)

  • Figure 1: Safety leaderboards of the 7 evaluated frontier models across four dimensions: Benchmark Evaluation, Adversarial Evaluation, Multilingual Evaluation, and Compliance Evaluation. (a) Language Safety Leaderboard; (b) Vision-Language Safety Leaderboard; (c) T2I Safety Leaderboard.
  • Figure 2: Safety Profiles of Evaluated Models. The radar charts depict the multidimensional safety characteristics of each model across Language and Vision–Language. Each axis corresponds to a normalized safety score (0--100%) along a specific evaluation dimension, including Benchmark, Adversarial, Multilingual, and Compliance (NIST, EU AI Act, FEAT) evaluations. Larger and more symmetric profiles indicate stronger and more balanced safety alignment.
  • Figure 3: Safety Profiles of Evaluated Models. The radar charts depict the multidimensional safety characteristics of Image Generation models. Each axis corresponds to a normalized safety score (0--100%) along a specific evaluation dimension, including Benchmark and Adversarial evaluations. Larger and more symmetric profiles indicate stronger and more balanced safety alignment.
  • Figure 4: Safe rate (%) of five models across five benchmarks.
  • Figure 5: Example unsafe responses across different safety benchmarks.
  • ...and 18 more figures