ChestX-Reasoner: Advancing Radiology Foundation Models with Reasoning through Step-by-Step Verification

TL;DR

ChestX-Reasoner addresses the gap in radiology AI by injecting step-by-step clinical reasoning into a radiology-focused multimodal model. It mines reasoning chains from clinical reports with GPT-4o and trains via a two-stage pipeline: supervised fine-tuning on reasoning-augmented and answer-only data, followed by reinforcement learning with a process reward to enforce reasoning quality. The work introduces RadRBench-CXR and RadRScore to jointly evaluate diagnostic outcomes and reasoning fidelity, reporting substantial improvements over both medical and general-domain baselines. All resources are open-sourced to accelerate research in medical reasoning for radiology AI.

Abstract

Recent advances in reasoning-enhanced large language models (LLMs) and multimodal LLMs (MLLMs) have significantly improved performance in complex tasks, yet medical AI models often overlook the structured reasoning processes inherent in clinical practice. In this work, we present ChestX-Reasoner, a radiology diagnosis MLLM designed to leverage process supervision mined directly from clinical reports, reflecting the step-by-step reasoning followed by radiologists. We construct a large dataset by extracting and refining reasoning chains from routine radiology reports. Our two-stage training framework combines supervised fine-tuning and reinforcement learning guided by process rewards to better align model reasoning with clinical standards. We introduce RadRBench-CXR, a comprehensive benchmark featuring 59K visual question answering samples with 301K clinically validated reasoning steps, and propose RadRScore, a metric evaluating reasoning factuality, completeness, and effectiveness. ChestX-Reasoner outperforms existing medical and general-domain MLLMs in both diagnostic accuracy and reasoning ability, achieving 16%, 5.9%, and 18% improvements in reasoning ability compared to the best medical MLLM, the best general MLLM, and its base model, respectively, as well as 3.3%, 24%, and 27% improvements in outcome accuracy. All resources are open-sourced to facilitate further research in medical reasoning MLLMs.

Figures (10)

• Figure 1: Mining Reasoning from Clinical Reports for Enhancing MLLMs. a. The reasoning workflow in radiologists' clinical report writing from low-level patterns of clinical observations to high-level disease identification. b. Mined reasoning samples from clinical reports in conjunction with GPT-4o. c. A two-stage training pipeline that incorporates process supervision to develop our reasoning-enhanced MLLM ChestX-Reasoner. d. An overview of evaluation results on RadRBench-CXR in terms of both reasoning ability and outcome accuracy, compared to extensive baseline models.
• Figure 1: Case 1. A case of binary disease diagnosis. \\ n and **xxx** in GPT-4o and Qwen2VL-72B's output represent a line break and bold text for xxx, respectively.
• Figure 2: Reasoning Mining Procedures and RadRScore Calculation a. Procedures of mining reasoning. We first construct reasoning plans by prompting GPT-4o based on given QA pairs and clinical reports, then extract relative clinical observations from the report as answers for each plan to derive the reasoning steps, and finally, refine the logic to form coherent and complete reasoning chains. b. RadRScore calculation targeting on factuality (the correctness of generated reasoning), completeness (the thoroughness in covering clinical findings), and effectiveness (the necessity and relevance of diagnostic processes). c. Human rating on the factuality, completeness, and effectiveness of our mined reasoning processes in the test set, where we use a five-point Likert scale ranging from -10 to 10, based on the provided question, ground truth answer, X-ray images, and clinical report.
• Figure 2: Case 2. A case of multiple disease diagnosis. \\ n and **xxx** in GPT-4o and Qwen2VL-72B's output represent a line break and bold text for xxx, respectively.
• Figure 3: Evaluation Results of Reasoning Ability. a. Question-answering pairs visualization of all task types. b. Reasoning abilities on dimensions of factuality, completeness, and effectiveness, as well as the averaged value, RadRScore.