LiveSecBench: A Dynamic and Event-Driven Safety Benchmark for Chinese Language Model Applications

TL;DR

LiveSecBench tackles the problem of rapidly evolving safety risks in Chinese LLM deployments by offering a dynamic benchmarking framework that combines automated adversarial prompt generation with human-in-the-loop QC. The framework includes seed-based data construction, seed fission for diversification, strategy-guided prompt synthesis, and automated judging, all under an ELO-based multi-dimensional evaluation. The v251215 release evaluates 57 LLMs across five safety dimensions, revealing that top closed-source systems currently outperform most open-source counterparts, while open-source models show strong performance in specific areas. The benchmark’s design emphasizes continuous updates, reproducibility, and real-world risk alignment, aiming to provide a robust, up-to-date safety standard for Chinese-language AI applications.

Abstract

We introduce LiveSecBench, a continuously updated safety benchmark specifically for Chinese-language LLM application scenarios. LiveSecBench constructs a high-quality and unique dataset through a pipeline that combines automated generation with human verification. By periodically releasing new versions to expand the dataset and update evaluation metrics, LiveSecBench provides a robust and up-to-date standard for AI safety. In this report, we introduce our second release v251215, which evaluates across five dimensions (Public Safety, Fairness & Bias, Privacy, Truthfulness, and Mental Health Safety.) We evaluate 57 representative LLMs using an ELO rating system, offering a leaderboard of the current state of Chinese LLM safety. The result is available at https://livesecbench.intokentech.cn/.

Figures (2)

• Figure 1: Overview of LiveSecBench-v251215 dataset distribution.
• Figure 2: Roadmap of LiveSecBench.