CSSBench: Evaluating the Safety of Lightweight LLMs against Chinese-Specific Adversarial Patterns

TL;DR

CSSBench addresses a critical gap in evaluating lightweight LLM safety for Chinese by introducing Chinese-specific adversarial patterns (Pinyin Mix, Homophones, Symbol Mix, Zero-width insertion), six real-world domains, three task formats, and an over-refusal evaluation set. The benchmark reveals that Chinese-specific obfuscations substantially degrade safety, with ASR around 31–39% and a CER in the low-to-mid 40s, and that over-refusal can mask actual risk. Through domain- and task-type analyses, CSSBench shows uneven protection across domains and that open-ended QA is particularly vulnerable. The work provides a practical framework and actionable insights for tuning safety policies in on-device Chinese LLMs and highlights the need for broader, multilingual safety evaluation beyond English-centric benchmarks.

Abstract

Large language models (LLMs) are increasingly deployed in cost-sensitive and on-device scenarios, and safety guardrails have advanced mainly in English. However, real-world Chinese malicious queries typically conceal intent via homophones, pinyin, symbol-based splitting, and other Chinese-specific patterns. These Chinese-specific adversarial patterns create the safety evaluation gap that is not well captured by existing benchmarks focused on English. This gap is particularly concerning for lightweight models, which may be more vulnerable to such specific adversarial perturbations. To bridge this gap, we introduce the Chinese-Specific Safety Benchmark (CSSBench) that emphasizes these adversarial patterns and evaluates the safety of lightweight LLMs in Chinese. Our benchmark covers six domains that are common in real Chinese scenarios, including illegal activities and compliance, privacy leakage, health and medical misinformation, fraud and hate, adult content, and public and political safety, and organizes queries into multiple task types. We evaluate a set of popular lightweight LLMs and measure over-refusal behavior to assess safety-induced performance degradation. Our results show that the Chinese-specific adversarial pattern is a critical challenge for lightweight LLMs. This benchmark offers a comprehensive evaluation of LLM safety in Chinese, assisting robust deployments in practice.

Figures (16)

• Figure 1:
• Figure 2: Examples of Pinyin Mix perturbations in CSSBench: full pinyin replacement, initial-letter abbreviation, and mixed character–pinyin forms.
• Figure 3: Examples of Homophone-based perturbations in CSSBench: huoxingwen, traditional Chinese replacements, and simplified homophone substitutions.
• Figure 4: Examples of Symbol Mix and Zero-width perturbations. The top rows show emoji and English mixing, while the bottom rows show a malicious sentence whose surface is unchanged but whose underlying token sequence is fragmented by zero-width characters.
• Figure 5: Comparison of Attack Success Rate (ASR) across Task Types. Models generally exhibit the highest ASR on open-ended QA tasks, indicating that free-form generation is the most vulnerable setting for safety compliance. Conversely, models achieve the lowest ASR on TF judgment tasks, representing their strongest safety performance under constrained formats. The results highlight how safety robustness varies significantly across different model families and task types.