Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MedRECT: A Medical Reasoning Benchmark for Error Correction in Clinical Texts

Naoto Iwase, Hiroki Okuyama, Junichiro Iwasawa

TL;DR

MedRECT presents the first cross-lingual benchmark for medical error detection, localization, and correction in Japanese and English, built via a scalable automated pipeline from JMLE and MEDEC. Evaluating nine contemporary LLMs across three tasks shows that reasoning-enabled models substantially outperform non-reasoning baselines, and that LoRA fine-tuning yields notable gains with cross-lingual transfer. The study reveals persistent cross-language gaps and demonstrates that bilingual reasoning data can improve English performance even when trained primarily on Japanese data, highlighting practical pathways toward safer multilingual medical AI. MedRECT thus provides a reproducible framework and resources to advance biomedical AI safety across languages and cultures.

Abstract

Large language models (LLMs) show increasing promise in medical applications, but their ability to detect and correct errors in clinical texts -- a prerequisite for safe deployment -- remains under-evaluated, particularly beyond English. We introduce MedRECT, a cross-lingual benchmark (Japanese/English) that formulates medical error handling as three subtasks: error detection, error localization (sentence extraction), and error correction. MedRECT is built with a scalable, automated pipeline from the Japanese Medical Licensing Examinations (JMLE) and a curated English counterpart, yielding MedRECT-ja (663 texts) and MedRECT-en (458 texts) with comparable error/no-error balance. We evaluate 9 contemporary LLMs spanning proprietary, open-weight, and reasoning families. Key findings: (i) reasoning models substantially outperform standard architectures, with up to 13.5% relative improvement in error detection and 51.0% in sentence extraction; (ii) cross-lingual evaluation reveals 5-10% performance gaps from English to Japanese, with smaller disparities for reasoning models; (iii) targeted LoRA fine-tuning yields asymmetric improvements in error correction performance (Japanese: +0.078, English: +0.168) while preserving reasoning capabilities; and (iv) our fine-tuned model exceeds human expert performance on structured medical error correction tasks. To our knowledge, MedRECT is the first comprehensive cross-lingual benchmark for medical error correction, providing a reproducible framework and resources for developing safer medical LLMs across languages.

MedRECT: A Medical Reasoning Benchmark for Error Correction in Clinical Texts

TL;DR

MedRECT presents the first cross-lingual benchmark for medical error detection, localization, and correction in Japanese and English, built via a scalable automated pipeline from JMLE and MEDEC. Evaluating nine contemporary LLMs across three tasks shows that reasoning-enabled models substantially outperform non-reasoning baselines, and that LoRA fine-tuning yields notable gains with cross-lingual transfer. The study reveals persistent cross-language gaps and demonstrates that bilingual reasoning data can improve English performance even when trained primarily on Japanese data, highlighting practical pathways toward safer multilingual medical AI. MedRECT thus provides a reproducible framework and resources to advance biomedical AI safety across languages and cultures.

Abstract

Large language models (LLMs) show increasing promise in medical applications, but their ability to detect and correct errors in clinical texts -- a prerequisite for safe deployment -- remains under-evaluated, particularly beyond English. We introduce MedRECT, a cross-lingual benchmark (Japanese/English) that formulates medical error handling as three subtasks: error detection, error localization (sentence extraction), and error correction. MedRECT is built with a scalable, automated pipeline from the Japanese Medical Licensing Examinations (JMLE) and a curated English counterpart, yielding MedRECT-ja (663 texts) and MedRECT-en (458 texts) with comparable error/no-error balance. We evaluate 9 contemporary LLMs spanning proprietary, open-weight, and reasoning families. Key findings: (i) reasoning models substantially outperform standard architectures, with up to 13.5% relative improvement in error detection and 51.0% in sentence extraction; (ii) cross-lingual evaluation reveals 5-10% performance gaps from English to Japanese, with smaller disparities for reasoning models; (iii) targeted LoRA fine-tuning yields asymmetric improvements in error correction performance (Japanese: +0.078, English: +0.168) while preserving reasoning capabilities; and (iv) our fine-tuned model exceeds human expert performance on structured medical error correction tasks. To our knowledge, MedRECT is the first comprehensive cross-lingual benchmark for medical error correction, providing a reproducible framework and resources for developing safer medical LLMs across languages.

Paper Structure

This paper contains 38 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmarks for Medical Reasoning
  4. Error Detection and Correction in Clinical Texts
  5. Japanese Medical NLP Resources
  6. Cross-Lingual Evaluation in Medicine
  7. MedRECT Dataset
  8. Task Definition
  9. Data Construction Pipeline
  10. MedRECT-ja Construction: Scalable Automated Pipeline
  11. Step 1: Data Synthesis
  12. Step 2: Quality Filtering
  13. Step 3: Model Deduplication
  14. MedRECT-en Construction: Building on Established Methodology
  15. Dataset Statistics
  16. ...and 23 more sections

Figures (2)

  • Figure 1: Examples from the MedRECT dataset showing different error types. Examples 1-2 show MedRECT-ja samples (translated to English for readability), while Example 3 shows a native MedRECT-en sample derived from MEDECabacha2025medecbenchmarkmedicalerror. Each example highlights the erroneous sentence (colored background) and provides the correct version.
  • Figure 2: Data construction pipeline for MedRECT benchmark creation. MedRECT-ja (top) transforms JMLE questions through automated synthesis, quality filtering, model deduplication, and LLM screening to produce 663 high-quality samples. MedRECT-en (bottom) applies identical LLM screening to the existing MEDEC MS Subset Test, yielding 458 samples. Red numbers indicate samples removed at each quality control step.