Benchmarking Ethical and Safety Risks of Healthcare LLMs in China-Toward Systemic Governance under Healthy China 2030

TL;DR

This study tackles the ethical and safety risks of healthcare LLMs in China under the Healthy China 2030 agenda. It introduces a 12,000-item Ethics & Safety QA benchmark spanning 11 ethics and 9 safety dimensions and evaluates SOTA Chinese medical LLMs (e.g., Qwen 2.5-32B, DeepSeek) in an offline, rubric-based setting. Baseline accuracy sits around 42.7%, with fine-tuning achieving up to 50.8% and reducing unsafe guidance and hallucinations, though certain areas like fairness/bias remain challenging. The authors diagnose systemic governance gaps and propose a practical framework—data ethics guidelines, safety simulation pipelines, and adaptive IRB processes—to enable safer, more transparent deployment of LLMs in healthcare aligned with Healthy China 2030.

Abstract

Large Language Models (LLMs) are poised to transform healthcare under China's Healthy China 2030 initiative, yet they introduce new ethical and patient-safety challenges. We present a novel 12,000-item Q&A benchmark covering 11 ethics and 9 safety dimensions in medical contexts, to quantitatively evaluate these risks. Using this dataset, we assess state-of-the-art Chinese medical LLMs (e.g., Qwen 2.5-32B, DeepSeek), revealing moderate baseline performance (accuracy 42.7% for Qwen 2.5-32B) and significant improvements after fine-tuning on our data (up to 50.8% accuracy). Results show notable gaps in LLM decision-making on ethics and safety scenarios, reflecting insufficient institutional oversight. We then identify systemic governance shortfalls-including the lack of fine-grained ethical audit protocols, slow adaptation by hospital IRBs, and insufficient evaluation tools-that currently hinder safe LLM deployment. Finally, we propose a practical governance framework for healthcare institutions (embedding LLM auditing teams, enacting data ethics guidelines, and implementing safety simulation pipelines) to proactively manage LLM risks. Our study highlights the urgent need for robust LLM governance in Chinese healthcare, aligning AI innovation with patient safety and ethical standards.

Figures (1)

• Figure 1: Medical Safety&Ethic Dataset.This Figure introduces the construction process, data distribution, and effectiveness of the medical ethics and safety dataset.We combined large language models such as DeepSeek-R1 and carried out manual quality control to construct a high-quality dataset from textbooks, policy documents, and laws and regulations. The dataset encompasses two major themes, namely medical ethics and medical safety, and is divided into 20 categories (Left). By using the constructed dataset to fine-tune Qwen2.5-32B, we found that its performance on the two medical ethics and safety evaluation sets, MedSafety and SafetyV2, outperforms that of the unfine-tuned model. Moreover, it also outperforms the unfine-tuned Qwen2.5-72B, while maintaining comparable performance in other medical tasks. (Right).Qwen2.5-32B+medical corpus and safety ethics data fine-tuning:Qwen2.5-32B fine-tuned on both general medical corpus and medical ethics & safety dataset.Qwen2.5-32B+safety ethics data fine-tuning:Qwen2.5-32B fine-tuned on medical ethics & safety dataset.