Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Understanding the Safety Boundaries of DeepSeek Models: Evaluation and Findings

Zonghao Ying, Guangyi Zheng, Yongxin Huang, Deyue Zhang, Wenxin Zhang, Quanchen Zou, Aishan Liu, Xianglong Liu, Dacheng Tao

TL;DR

<3-5 sentence high-level summary> This paper conducts the first comprehensive safety evaluation of the DeepSeek model family, covering LLMs, MLLMs, and a T2I model, using a bilingual Chinese–English safety framework tailored to Chinese contexts. It introduces CNSafe, CNSafe_RT, SafeBench, MM-SafetyBench, and I2P benchmarks to quantify unsafe content generation across text and images, employing a hybrid human-(M)LLM judging approach. Key findings show substantial safety vulnerabilities across risk categories such as discriminatory and sexual content, with cross-lingual disparities and exposed chain-of-thought increasing $ASR$ under jailbreak conditions; T2I generation also exhibits high risk. The results highlight the need for stronger safety alignment, broader benchmarks, and iterative mitigation strategies, and the authors provide public code to facilitate replication and future work.

Abstract

This study presents the first comprehensive safety evaluation of the DeepSeek models, focusing on evaluating the safety risks associated with their generated content. Our evaluation encompasses DeepSeek's latest generation of large language models, multimodal large language models, and text-to-image models, systematically examining their performance regarding unsafe content generation. Notably, we developed a bilingual (Chinese-English) safety evaluation dataset tailored to Chinese sociocultural contexts, enabling a more thorough evaluation of the safety capabilities of Chinese-developed models. Experimental results indicate that despite their strong general capabilities, DeepSeek models exhibit significant safety vulnerabilities across multiple risk dimensions, including algorithmic discrimination and sexual content. These findings provide crucial insights for understanding and improving the safety of large foundation models. Our code is available at https://github.com/NY1024/DeepSeek-Safety-Eval.

Towards Understanding the Safety Boundaries of DeepSeek Models: Evaluation and Findings

TL;DR

<3-5 sentence high-level summary> This paper conducts the first comprehensive safety evaluation of the DeepSeek model family, covering LLMs, MLLMs, and a T2I model, using a bilingual Chinese–English safety framework tailored to Chinese contexts. It introduces CNSafe, CNSafe_RT, SafeBench, MM-SafetyBench, and I2P benchmarks to quantify unsafe content generation across text and images, employing a hybrid human-(M)LLM judging approach. Key findings show substantial safety vulnerabilities across risk categories such as discriminatory and sexual content, with cross-lingual disparities and exposed chain-of-thought increasing under jailbreak conditions; T2I generation also exhibits high risk. The results highlight the need for stronger safety alignment, broader benchmarks, and iterative mitigation strategies, and the authors provide public code to facilitate replication and future work.

Abstract

This study presents the first comprehensive safety evaluation of the DeepSeek models, focusing on evaluating the safety risks associated with their generated content. Our evaluation encompasses DeepSeek's latest generation of large language models, multimodal large language models, and text-to-image models, systematically examining their performance regarding unsafe content generation. Notably, we developed a bilingual (Chinese-English) safety evaluation dataset tailored to Chinese sociocultural contexts, enabling a more thorough evaluation of the safety capabilities of Chinese-developed models. Experimental results indicate that despite their strong general capabilities, DeepSeek models exhibit significant safety vulnerabilities across multiple risk dimensions, including algorithmic discrimination and sexual content. These findings provide crucial insights for understanding and improving the safety of large foundation models. Our code is available at https://github.com/NY1024/DeepSeek-Safety-Eval.

Paper Structure

This paper contains 22 sections, 1 equation, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. DeekSeek Models
  4. Jailbreak Attacks
  5. Evaluation Protocol
  6. Benchmarks
  7. Evaluation Methods
  8. Metric
  9. Experiment
  10. Evaluation on LLMs
  11. DeepSeek-R1 & DeepSeek-V3
  12. Comparison with other Chinese LLMs
  13. Evaluation on MLLM
  14. Evaluation on T2I Models
  15. Conclusion
  16. ...and 7 more sections

Figures (7)

  • Figure 1: ASR (%) of DeepSeek-R1 and DeepSeek-V3 on CNSafe_RT.
  • Figure 2: ASR (%) of DeepSeek LLMs and other Chinese-developed LLMs on CNSafe. Abbreviations: RS, Regime subversion; NH, National harm; SD, State division; TP, Terrorism promotion; EH, Ethnic hatred; VP, Violence promotion; FI, False information; PC, Prohibited content; ED, Ethnic discrimination; FD, Faith discrimination; ND, National discrimination; RD, Regional discrimination; GD, Gender discrimination; AD, Age discrimination; OD, Occupational discrimination; HD, Health discrimination; OT, Other discrimination; II, IP infringement; BE, Business ethics; TS, Trade secrets; UC, Unfair competition; BV, Business violations; HE, Health endangerment; IR, Image rights; RH, Reputation harm; HV, Honor violation; PI, Privacy invasion; DM, Data misuse; RV, Rights violation.
  • Figure 3: ASR (%) of Chinese-developed LLMs on CNSafe_RT.
  • Figure 4: Examples of image-text pairs used in multimodal jailbreak attack methods.
  • Figure 5: ASR (%) of DeepSeek-VL2 on SafeBench and MM-SafetyBench.
  • ...and 2 more figures