CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMs

TL;DR

CARES introduces a comprehensive medical LLM safety benchmark comprising over 18K prompts across eight safety principles, four harm levels, and four prompting styles to evaluate harmful content, jailbreak vulnerability, and false refusals. It defines a three-way response taxonomy (Accept, Caution, Refuse) and a fine-grained Safety Score $SS = \frac{1}{N} \sum_{i=1}^{N} \mathrm{score}(h_i, a_i)$ to rate model behavior, capturing both unsafe accepts and cautious responses. The study reveals that state-of-the-art models remain vulnerable to jailbreak-style prompt rewrites and may over-refuse benign queries, while domain-tuned models show competitive safety performance; jailbreak-awareness coupled with a reminder-based conditioning strategy improves safety across models. The work provides a practical mitigation by training a jailbreak-detection classifier that guides models via reminders, establishing a foundation for more robust and trustworthy medical AI systems.

Abstract

Large language models (LLMs) are increasingly deployed in medical contexts, raising critical concerns about safety, alignment, and susceptibility to adversarial manipulation. While prior benchmarks assess model refusal capabilities for harmful prompts, they often lack clinical specificity, graded harmfulness levels, and coverage of jailbreak-style attacks. We introduce CARES (Clinical Adversarial Robustness and Evaluation of Safety), a benchmark for evaluating LLM safety in healthcare. CARES includes over 18,000 prompts spanning eight medical safety principles, four harm levels, and four prompting styles: direct, indirect, obfuscated, and role-play, to simulate both malicious and benign use cases. We propose a three-way response evaluation protocol (Accept, Caution, Refuse) and a fine-grained Safety Score metric to assess model behavior. Our analysis reveals that many state-of-the-art LLMs remain vulnerable to jailbreaks that subtly rephrase harmful prompts, while also over-refusing safe but atypically phrased queries. Finally, we propose a mitigation strategy using a lightweight classifier to detect jailbreak attempts and steer models toward safer behavior via reminder-based conditioning. CARES provides a rigorous framework for testing and improving medical LLM safety under adversarial and ambiguous conditions.

Figures (17)

• Figure 1: Overview of the CARES dataset construction pipeline. We begin by mining safety rules from clinical guidelines (e.g., AMA ethics, HIPAA, Constitutional AI, and prior safety rulebases), which are distilled into 8 medical safety principles. Prompts are generated across 4 harm levels (0–3) using strong LLMs and validated by humans. Each prompt is then adversarially rewritten using three jailbreak strategies—indirect, obfuscation, and role-play—to evaluate model robustness under adversarial disguise. This yields both direct and jailbroken variants, resulting in the final CARES-18K benchmark.
• Figure 2: Pearson correlation agreement between the models and human raters. "HumanVote" refers to the aggregated rating obtained via majority vote across the five human annotations.
• Figure 3: Ranking agreement (averaged across all samples) between the model, Rater 1, and Rater 2, evaluated using multiple correlation metrics. Higher values indicate better agreement for all metrics except MSE, where lower is better.
• Figure 4: Distribution of CARES, along multiple dimensions, such as prompt generation model, harmfulness level, jailbreak strategy and safety principle, are demonstrated
• Figure 5: Performance comparison of language models on the 9K safety test dataset. Metrics shown include Safety Score (SS), Accuracy (ACC), and F1 score (F1).