A Real-World Evaluation of LLM Medication Safety Reviews in NHS Primary Care

TL;DR

This study provides the first real-world evaluation of an LLM-based medication safety review system applied to NHS primary care data, addressing the gap between benchmark success and practical safety outcomes. Using a population-scale EHR from NHS Cheshire and Merseyside, the authors deploy a 120B OpenAI model in a structured, three-level hierarchical evaluation framework and a detailed failure taxonomy to characterize performance across clinical complexity. While achieving 100% sensitivity in identifying issues, the system correctly identifies all issues and interventions in only 46.9% of cases, revealing a dominant contextual reasoning failure pattern over factual errors. The findings challenge the emphasis on knowledge augmentation alone, highlighting the need for better uncertainty calibration, agentic information gathering, and implicit healthcare delivery knowledge to enable safe deployment, and they call for larger, prospective evaluations across diverse clinical contexts. The work has practical implications for designing safer clinical AI workflows, particularly in medication safety, by prioritizing robust contextual reasoning and human-in-the-loop strategies over purely knowledge-retrieval improvements.

Abstract

Large language models (LLMs) often match or exceed clinician-level performance on medical benchmarks, yet very few are evaluated on real clinical data or examined beyond headline metrics. We present, to our knowledge, the first evaluation of an LLM-based medication safety review system on real NHS primary care data, with detailed characterisation of key failure behaviours across varying levels of clinical complexity. In a retrospective study using a population-scale EHR spanning 2,125,549 adults in NHS Cheshire and Merseyside, we strategically sampled patients to capture a broad range of clinical complexity and medication safety risk, yielding 277 patients after data-quality exclusions. An expert clinician reviewed these patients and graded system-identified issues and proposed interventions. Our primary LLM system showed strong performance in recognising when a clinical issue is present (sensitivity 100\% [95\% CI 98.2--100], specificity 83.1\% [95\% CI 72.7--90.1]), yet correctly identified all issues and interventions in only 46.9\% [95\% CI 41.1--52.8] of patients. Failure analysis reveals that, in this setting, the dominant failure mechanism is contextual reasoning rather than missing medication knowledge, with five primary patterns: overconfidence in uncertainty, applying standard guidelines without adjusting for patient context, misunderstanding how healthcare is delivered in practice, factual errors, and process blindness. These patterns persisted across patient complexity and demographic strata, and across a range of state-of-the-art models and configurations. We provide 45 detailed vignettes that comprehensively cover all identified failure cases. This work highlights shortcomings that must be addressed before LLM-based clinical AI can be safely deployed. It also begs larger-scale, prospective evaluations and deeper study of LLM behaviours in clinical contexts.

Figures (6)

• Figure 1: Evaluation framework and primary findings.(a) Evaluation workflow: patient profiles are processed by the System; a clinician reviews outputs marking issues and interventions as correct (✓), partially correct ($\approx$), or incorrect (×). (b) Hierarchical performance across 277 patients at three stages: binary classification (Level 1), issue correctness (Level 2), and intervention appropriateness (Level 3). (c) Distribution of 178 failure instances by failure reason and evaluation stage.
• Figure 2: Taxonomy and examples of System failures. Classification of 178 failure instances across 148 patients into five failure reasons and their corresponding failure modes. Representative vignettes for each failure type showing the clinical scenario, System output, and clinician assessment. Ticks indicate correct identifications; crosses indicate errors. A full set of vignettes can be found in Appendix \ref{['app:failure-vignettes']}.
• Figure 3: Correlation matrix of patient complexity variables and clinician score. Age, medication count, and QoF register count are highly intercorrelated ($r = 0.56$--$0.67$, blue), indicating these metrics capture overlapping aspects of patient complexity. Each shows similar bivariate association with performance ($r = -0.25$ to $-0.28$, red). Multiple regression confirmed no variable has significant independent predictive value when controlling for the others ($R^2 = 0.10$; all partial $r < 0.12$, all $p > 0.05$), suggesting a single latent complexity construct underlies the observed performance decline.
• Figure 4: Multi-model performance comparison across six configurations. GPT-OSS-120B-medium achieved highest performance (0.459), with clear performance degradation for smaller scale within architecture (GPT-OSS-20B: 0.334, -37.4%). Gemma architecture models substantially underperformed despite larger parameter counts (Gemma 3 27B: 0.196, MedGemma 27B: 0.239), with even the 20B GPT-OSS model outperforming the 27B Gemma models by 39.8-70.3%. Error bars show standard error of the mean.
• Figure 5: the System's performance by indicator. Mean clinician scores shown with standard error bars. Sample sizes per filter range from 6-9 patients (only cases where clinician validated intervention was needed). Eight filters shown after excluding filters 16 and 43 due to implementation errors.