Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models

TL;DR

This work presents Chinese SafetyQA, the first short-form factuality benchmark focused on safety knowledge in the Chinese context (law, policy, ethics). Encompassing 2,000 QA pairs and 2,000 MCQ items across seven safety domains with 103 subtopics, it enables efficient evaluation of LLMs’ safety factuality. Evaluation across 50+ models reveals pervasive factuality gaps, overconfidence, and a Tip-of-the-Tongue effect, with Retrieval-Augmented Generation offering notable gains—especially for smaller models—while self-reflection yields limited benefits. The benchmark’s design emphasizes static, harmless, and bilingual-ready content, supporting robust model deployment in China and guiding future enhancements toward multi-modal safety knowledge.

Abstract

With the rapid advancement of Large Language Models (LLMs), significant safety concerns have emerged. Fundamentally, the safety of large language models is closely linked to the accuracy, comprehensiveness, and clarity of their understanding of safety knowledge, particularly in domains such as law, policy and ethics. This factuality ability is crucial in determining whether these models can be deployed and applied safely and compliantly within specific regions. To address these challenges and better evaluate the factuality ability of LLMs to answer short questions, we introduce the Chinese SafetyQA benchmark. Chinese SafetyQA has several properties (i.e., Chinese, Diverse, High-quality, Static, Easy-to-evaluate, Safety-related, Harmless). Based on Chinese SafetyQA, we perform a comprehensive evaluation on the factuality abilities of existing LLMs and analyze how these capabilities relate to LLM abilities, e.g., RAG ability and robustness against attacks.

Figures (14)

• Figure 1: Chinese SafetyQA has three levels of classification, covering seven different security domains, with a total of 103 subtopics, capable of comprehensively addressing the risk knowledge in various domains. The description of abbreviations can be found in Appendix \ref{['appendix:list-of-abbr.']}.
• Figure 2: Data Processing Workflow Diagram
• Figure 3: Average accuracy (%) for each confidence bucket. Confidence scores are divided into bins ranging from 0 to 100 in 5-point intervals. Each entry represents the mean accuracy of predictions within the corresponding confidence range.
• Figure 4: The effect of self-reflection strategy.
• Figure 5: The effect of different RAG strategies, including: no RAG, active RAG, passive RAG.