Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RxSafeBench: Identifying Medication Safety Issues of Large Language Models in Simulated Consultation

Jiahao Zhao, Luxin Xu, Minghuan Tan, Lichao Zhang, Ahmadreza Argha, Hamid Alinejad-Rokny, Min Yang

TL;DR

RxSafeBench tackles the lack of real-world medication-safety evaluation data by constructing RxRisk DB with over 35,000 structured entries and simulating department-specific clinical consultations. It combines a two-stage filtering pipeline and GPT-4 automated scoring to produce 2,443 high-quality, three-option MCQ scenarios that probe contraindication and drug-interaction reasoning. Experimental results show current LLMs struggle to reason about safety when risks are implicit, with varying performance across specialties and a clear need for prompting strategies and domain tuning. The work establishes a foundational benchmark for medication safety in AI-assisted clinical decision support and aims to foster safer, more trustworthy medical AI systems.

Abstract

Numerous medical systems powered by Large Language Models (LLMs) have achieved remarkable progress in diverse healthcare tasks. However, research on their medication safety remains limited due to the lack of real world datasets, constrained by privacy and accessibility issues. Moreover, evaluation of LLMs in realistic clinical consultation settings, particularly regarding medication safety, is still underexplored. To address these gaps, we propose a framework that simulates and evaluates clinical consultations to systematically assess the medication safety capabilities of LLMs. Within this framework, we generate inquiry diagnosis dialogues with embedded medication risks and construct a dedicated medication safety database, RxRisk DB, containing 6,725 contraindications, 28,781 drug interactions, and 14,906 indication-drug pairs. A two-stage filtering strategy ensures clinical realism and professional quality, resulting in the benchmark RxSafeBench with 2,443 high-quality consultation scenarios. We evaluate leading open-source and proprietary LLMs using structured multiple choice questions that test their ability to recommend safe medications under simulated patient contexts. Results show that current LLMs struggle to integrate contraindication and interaction knowledge, especially when risks are implied rather than explicit. Our findings highlight key challenges in ensuring medication safety in LLM-based systems and provide insights into improving reliability through better prompting and task-specific tuning. RxSafeBench offers the first comprehensive benchmark for evaluating medication safety in LLMs, advancing safer and more trustworthy AI-driven clinical decision support.

RxSafeBench: Identifying Medication Safety Issues of Large Language Models in Simulated Consultation

TL;DR

RxSafeBench tackles the lack of real-world medication-safety evaluation data by constructing RxRisk DB with over 35,000 structured entries and simulating department-specific clinical consultations. It combines a two-stage filtering pipeline and GPT-4 automated scoring to produce 2,443 high-quality, three-option MCQ scenarios that probe contraindication and drug-interaction reasoning. Experimental results show current LLMs struggle to reason about safety when risks are implicit, with varying performance across specialties and a clear need for prompting strategies and domain tuning. The work establishes a foundational benchmark for medication safety in AI-assisted clinical decision support and aims to foster safer, more trustworthy medical AI systems.

Abstract

Numerous medical systems powered by Large Language Models (LLMs) have achieved remarkable progress in diverse healthcare tasks. However, research on their medication safety remains limited due to the lack of real world datasets, constrained by privacy and accessibility issues. Moreover, evaluation of LLMs in realistic clinical consultation settings, particularly regarding medication safety, is still underexplored. To address these gaps, we propose a framework that simulates and evaluates clinical consultations to systematically assess the medication safety capabilities of LLMs. Within this framework, we generate inquiry diagnosis dialogues with embedded medication risks and construct a dedicated medication safety database, RxRisk DB, containing 6,725 contraindications, 28,781 drug interactions, and 14,906 indication-drug pairs. A two-stage filtering strategy ensures clinical realism and professional quality, resulting in the benchmark RxSafeBench with 2,443 high-quality consultation scenarios. We evaluate leading open-source and proprietary LLMs using structured multiple choice questions that test their ability to recommend safe medications under simulated patient contexts. Results show that current LLMs struggle to integrate contraindication and interaction knowledge, especially when risks are implied rather than explicit. Our findings highlight key challenges in ensuring medication safety in LLM-based systems and provide insights into improving reliability through better prompting and task-specific tuning. RxSafeBench offers the first comprehensive benchmark for evaluating medication safety in LLMs, advancing safer and more trustworthy AI-driven clinical decision support.

Paper Structure

This paper contains 17 sections, 5 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. MATERIALS AND METHODS
  3. Collection of Medication Information
  4. Online Medication Extraction
  5. Medication Categorization
  6. Consultation Simulation
  7. Department-Specific Instruction
  8. Incorporating Medication Risks into Dialogue Generation
  9. Question Construction
  10. Two-stage filtering strategy
  11. Indication-Based Filtering and Sampling
  12. Automatic Scoring Mechanism
  13. EXPERIMENTS AND ANALYSIS
  14. Experiment Setup
  15. Dataset
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Overview of the RxSafeBench Framework.
  • Figure 2: An example with medication contraindication.
  • Figure 3: Statistical test results for model answer evaluation: (a,b) Chi-square test results of three types of scores and model answer selection; (c,d) t-test results of question length and rounds versus model answer accuracy.