Importance of Prompt Optimisation for Error Detection in Medical Notes Using Language Models

TL;DR

It is shown that automatic prompt optimisation with Genetic-Pareto (GEPA) improves error detection over the baseline accuracy performance from 0.669 to 0.785 with GPT-5 and 0.690 with Qwen3-32B, approaching the performance of medical doctors and achieving state-of-the-art performance on the MEDEC benchmark dataset.

Abstract

Errors in medical text can cause delays or even result in incorrect treatment for patients. Recently, language models have shown promise in their ability to automatically detect errors in medical text, an ability that has the opportunity to significantly benefit healthcare systems. In this paper, we explore the importance of prompt optimisation for small and large language models when applied to the task of error detection. We perform rigorous experiments and analysis across frontier language models and open-source language models. We show that automatic prompt optimisation with Genetic-Pareto (GEPA) improves error detection over the baseline accuracy performance from 0.669 to 0.785 with GPT-5 and 0.578 to 0.690 with Qwen3-32B, approaching the performance of medical doctors and achieving state-of-the-art performance on the MEDEC benchmark dataset. Code available on GitHub: https://github.com/CraigMyles/clinical-note-error-detection

Figures (9)

• Figure 1: Diagram highlighting our experiment workflow, showing that we utilise the MS training and validation subsets from the MEDEC dataset ben-abacha-etal-2025-medec for prompt optimisation with GEPA agrawal2025gepa. The data contains medical text (indicated by the Report text) with binary labels indicating whether or not the text contains an error. The MS and UW test sets are used for evaluation only. We evaluate seven different models, including the GPT-5 frontier model openai2025gpt5 and various open-source Qwen3 models qwen3, in 28 different configurations.
• Figure 2: Illustrative example from the MS-Train dataset, selected from the shortest 1% of erroneous examples in the dataset. The narrative contains one injected diagnostic medical error (sentence 2). We show the corresponding correction for clarity.
• Figure 3: Distribution of sample categories across MEDEC dataset splits.
• Figure 4: Performance of different inference--reflector pairs using GEPA optimisation on the MEDEC-MS validation set. a) shows the absolute performance. b) shows the difference between the base prompt and optimised prompt performance.
• Figure 5: Benchmark P1 accuracy on the combined MEDEC test sets, with MS+UW weighted by their respective sample counts; comparing previously benchmarked ben-abacha-etal-2025-medecsaeed-2024-medifact-mediqa (orange) vs our P1 benchmarks (blue) as well as GEPA optimised.