JMedEthicBench: A Multi-Turn Conversational Benchmark for Evaluating Medical Safety in Japanese Large Language Models

TL;DR

JMedEthicBench addresses the need for robust medical safety evaluation of Japanese LLMs by introducing a multi-turn benchmark grounded in 67 JMA guidelines and generated via automated jailbreak strategies, yielding over $5\times 10^4$ conversations. The study employs a dual-scorer protocol to rate safety across 27 models, revealing that commercial models generally maintain higher safety, while medical-specialized models are more vulnerable and safety erodes across turns, with a median drop from $9.5$ to $5.0$ ($p < 0.001$). Cross-lingual analyses show vulnerabilities persist across Japanese and English, suggesting fundamental alignment limitations rather than language-specific issues. The work underscores the need for domain-aware safety training during domain adaptation and provides a scalable data-generation and evaluation framework to advance reproducible safety research in medical AI.

Abstract

As Large Language Models (LLMs) are increasingly deployed in healthcare field, it becomes essential to carefully evaluate their medical safety before clinical use. However, existing safety benchmarks remain predominantly English-centric, and test with only single-turn prompts despite multi-turn clinical consultations. To address these gaps, we introduce JMedEthicBench, the first multi-turn conversational benchmark for evaluating medical safety of LLMs for Japanese healthcare. Our benchmark is based on 67 guidelines from the Japan Medical Association and contains over 50,000 adversarial conversations generated using seven automatically discovered jailbreak strategies. Using a dual-LLM scoring protocol, we evaluate 27 models and find that commercial models maintain robust safety while medical-specialized models exhibit increased vulnerability. Furthermore, safety scores decline significantly across conversation turns (median: 9.5 to 5.0, $p < 0.001$). Cross-lingual evaluation on both Japanese and English versions of our benchmark reveals that medical model vulnerabilities persist across languages, indicating inherent alignment limitations rather than language-specific factors. These findings suggest that domain-specific fine-tuning may accidentally weaken safety mechanisms and that multi-turn interactions represent a distinct threat surface requiring dedicated alignment strategies.

Figures (6)

• Figure 1: Safety-helpfulness scatter plot of 24 evaluated models on JMedEthicBench. Safety pass rate is calculated as the percentage of conversations where any turn scores above 2 (indicating safe behavior), and helpfulness scores are based on 287 questions from the Japanese National Medical Licensing Examination 2025.
• Figure 2: Overview of the JMedEthicBench construction pipeline. Step 1: We generate 3,350 harmful requests from 67 detailed JMA guidelines with concrete clinical scenarios using 5 LLMs via prompt engineering. Step 2: After filtering to 1,935 validated requests, we employ an auto-generation workflow to discover 7 distinct jailbreak strategies and produce 54,180 multi-turn harmful prompts. Step 3: Through hierarchical sampling, we create a test set of 2,345 instances for evaluating LLM safety.
• Figure 3: Safety scores across conversation turns for all evaluated models. General models (left) and medical models (right) are separated by the dashed line. Commercial models (e.g., Claude, GPT-5) maintain high scores across all turns, while medical-specialized models exhibit pronounced score degradation in later turns.
• Figure 4: Safety score distribution across conversation turns. Scores decline significantly from Turn 0 (median $\approx$ 9.5) to Turn 2 (median $\approx$ 5.0). Statistical significance is indicated by *** ($p < 0.001$, Mann-Whitney U test with Bonferroni correction). In the boxplot, the orange line represents the median, the green dashed line indicates the mean, the box spans the interquartile range (IQR, 25th--75th percentiles), whiskers extend to 1.5$\times$IQR, and points beyond the whiskers indicate outliers.
• Figure 5: Safety score distribution by jailbreak strategy. All seven strategies produce similar score distributions, suggesting that models lack robust defenses against diverse attack patterns. In the boxplot, the orange line represents the median, the green dashed line indicates the mean, the box spans the IQR, whiskers extend to 1.5$\times$IQR, and points beyond the whiskers indicate outliers.