MedErrBench: A Fine-Grained Multilingual Benchmark for Medical Error Detection and Correction with Clinical Expert Annotations

TL;DR

MedErrBench addresses a critical gap in clinical NLP by delivering the first fine-grained multilingual benchmark for medical error detection, localization, and correction across English, Chinese, and Arabic, guided by a clinician-defined taxonomy of ten error types. The methodology combines careful multilingual data construction (not mere translations), error injection, and rich annotations (difficulty and reasoning type), followed by extensive evaluation of diverse model families and prompting configurations. Key findings show notable performance gaps in non-English settings, with localization and correction proving harder than detection, and highlight the benefits of definitions and few-shot exemplars while also revealing model-specific sensitivities to prompts. By publicly releasing MedErrBench and its evaluation protocols, the work advances multilingual clinical NLP toward safer, more equitable AI-enabled healthcare globally, while laying groundwork for richer annotations (severity, equity) and future expansion.

Abstract

Inaccuracies in existing or generated clinical text may lead to serious adverse consequences, especially if it is a misdiagnosis or incorrect treatment suggestion. With Large Language Models (LLMs) increasingly being used across diverse healthcare applications, comprehensive evaluation through dedicated benchmarks is crucial. However, such datasets remain scarce, especially across diverse languages and contexts. In this paper, we introduce MedErrBench, the first multilingual benchmark for error detection, localization, and correction, developed under the guidance of experienced clinicians. Based on an expanded taxonomy of ten common error types, MedErrBench covers English, Arabic and Chinese, with natural clinical cases annotated and reviewed by domain experts. We assessed the performance of a range of general-purpose, language-specific, and medical-domain language models across all three tasks. Our results reveal notable performance gaps, particularly in non-English settings, highlighting the need for clinically grounded, language-aware systems. By making MedErrBench and our evaluation protocols publicly-available, we aim to advance multilingual clinical NLP to promote safer and more equitable AI-based healthcare globally. The dataset is available in the supplementary material. An anonymized version of the dataset is available at: https://github.com/congboma/MedErrBench.

Figures (15)

• Figure 1: Overview of MedErrBench.
• Figure 2: Distribution of difficulty level and reasoning type.
• Figure 3: Performance comparison across easy, medium, and hard examples in few-shot learning.
• Figure 4: Comparison of models based on knowledge-based and description-based evaluation.
• Figure S1: Distribution of Error Types by Language