CorBenchX: Large-Scale Chest X-Ray Error Dataset and Vision-Language Model Benchmark for Report Error Correction

TL;DR

CorBenchX addresses the lack of large-scale benchmarks for automated error detection and correction in chest X-ray reports. It constructs a 26,326-sample dataset by injecting clinically plausible errors into MIMIC-CXR reports using DeepSeek-R1 and benchmarks nine vision-language systems under zero-shot prompting, revealing that even top systems achieve limited clinical-grade performance. To close the gap, the authors propose Multi-Step Reinforcement Learning (MSRL) with three stages—error identification, description, and correction—and use GRPO to optimize for format compliance, error-type accuracy, and textual fidelity, yielding substantial gains (e.g., 38.3% precision improvement in single-error detection). The work demonstrates the feasibility of a unified, scalable benchmark and provides a concrete path toward more reliable automated radiology report correction, while acknowledging current limitations and outlining directions to incorporate multi-modal data and longitudinal clinical context.

Abstract

AI-driven models have shown great promise in detecting errors in radiology reports, yet the field lacks a unified benchmark for rigorous evaluation of error detection and further correction. To address this gap, we introduce CorBenchX, a comprehensive suite for automated error detection and correction in chest X-ray reports, designed to advance AI-assisted quality control in clinical practice. We first synthesize a large-scale dataset of 26,326 chest X-ray error reports by injecting clinically common errors via prompting DeepSeek-R1, with each corrupted report paired with its original text, error type, and human-readable description. Leveraging this dataset, we benchmark both open- and closed-source vision-language models,(e.g., InternVL, Qwen-VL, GPT-4o, o4-mini, and Claude-3.7) for error detection and correction under zero-shot prompting. Among these models, o4-mini achieves the best performance, with 50.6 % detection accuracy and correction scores of BLEU 0.853, ROUGE 0.924, BERTScore 0.981, SembScore 0.865, and CheXbertF1 0.954, remaining below clinical-level accuracy, highlighting the challenge of precise report correction. To advance the state of the art, we propose a multi-step reinforcement learning (MSRL) framework that optimizes a multi-objective reward combining format compliance, error-type accuracy, and BLEU similarity. We apply MSRL to QwenVL2.5-7B, the top open-source model in our benchmark, achieving an improvement of 38.3% in single-error detection precision and 5.2% in single-error correction over the zero-shot baseline.

Figures (5)

• Figure 1: Overview of CorBenchX. (a): Error report dataset construction pipeline and dataset statistics. (b): Benchmark results across nine vision–language models for error detection and correction. (c): Illustration of our proposed multi-step reinforcement learning (MsRL) method and its performance improvements over the baseline.
• Figure 2: Example of a chest X-ray, paired original radiology report, and the corresponding error-injected report with labels. Text spans highlighted in red denote the injected errors, while the corrected spans are shown in green.
• Figure 3: Illustration of our multi-step reinforcement-learning framework: the model sequentially performs error identification, description, and correction, with each stage guided by a tailored reward.
• Figure 4: Precision and recall for single-error detection across various VLMs and models enhanced by our MSRL, broken down by the five error categories.
• Figure 5: Precision and recall for multi-error detection across various VLMs and models enhanced by our MSRL, broken down by the five error categories.