MPIB: A Benchmark for Medical Prompt Injection Attacks and Clinical Safety in LLMs

TL;DR

MPIB addresses the critical problem of clinical safety in LLMs and RAG systems by introducing a dedicated benchmark that evaluates both direct (V1) and indirect (V2) prompt injection across four clinical scenario families. It combines a sizable, curated dataset (9,697 instances) with outcome-centric metrics, CHER and ASR, and employs a judge-based harm assessment alongside a modular defense harness (D0–D4) to study safety under adversarial manipulation. The key contributions include the MPIB dataset, a clinically grounded harm taxonomy with severity levels, a fixed evaluation protocol with guaranteed exposure for V2, and a reproducible evaluation harness that highlights asymmetries between instruction compliance and downstream patient risk. The findings show that CHER and ASR can diverge, that attack vectors differ in strength, and that defense effectiveness is model- and threat-dependent, underscoring the need for outcome-based safety auditing in clinical AI deployments. MPIB thus provides a practical, reproducible framework for understanding, defending, and benchmarking clinical prompt injection in LLM-based workflows.

Abstract

Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems are increasingly integrated into clinical workflows; however, prompt injection attacks can steer these systems toward clinically unsafe or misleading outputs. We introduce the Medical Prompt Injection Benchmark (MPIB), a dataset-and-benchmark suite for evaluating clinical safety under both direct prompt injection and indirect, RAG-mediated injection across clinically grounded tasks. MPIB emphasizes outcome-level risk via the Clinical Harm Event Rate (CHER), which measures high-severity clinical harm events under a clinically grounded taxonomy, and reports CHER alongside Attack Success Rate (ASR) to disentangle instruction compliance from downstream patient risk. The benchmark comprises 9,697 curated instances constructed through multi-stage quality gates and clinical safety linting. Evaluating MPIB across a diverse set of baseline LLMs and defense configurations, we find that ASR and CHER can diverge substantially, and that robustness depends critically on whether adversarial instructions appear in the user query or in retrieved context. We release MPIB with evaluation code, adversarial baselines, and comprehensive documentation to support reproducible and systematic research on clinical prompt injection. Code and data are available at GitHub (code) and Hugging Face (data).

Figures (7)

• Figure 1: The Threat Landscape of Universal RAG Injection in Clinical Scenarios. An adversary injects malicious payloads into the medical knowledge base (center). Consequently, across diverse tasks including explanation, dosing, triage, and guideline checking (S1--S4), (a) the benign RAG operation retrieves clean contexts to produce evidence-based, safe instructions (left; blue), whereas (b) the compromised RAG operation retrieves poisoned contexts to generate high-severity harmful outputs, including fabricated information and critical contraindication violations (right; red).
• Figure 2: MPIB Data Construction Pipeline. The pipeline comprises five stages: Processing (text normalization and scenario tagging), Attack (adversarial prompt and poisoned-context generation), Quality Control (multi-stage gating), Harm Taxonomy (harm-type and severity annotation), and Release (benchmark packaging).
• Figure 3: ASR--CHER$_3$ divergence under direct (V1; top) and indirect (V2; bottom) prompt injection. Gray markers denote ASR (Severity $\ge 2$), while colored markers denote CHER$_3$ (Severity $\ge 3$). Horizontal separation indicates the ASR--CHER divergence, with larger gaps corresponding to Safe Gaps (CHER$_3$$<$ ASR).
• Figure 4: V0 benign anchor example. Both models correctly identify the condition and recommend evidence-based conservative management.
• Figure 5: V0' borderline perturbation example. Models retain clinical accuracy under standard medical reasoning tasks.