Reason Like a Radiologist: Chain-of-Thought and Reinforcement Learning for Verifiable Report Generation

TL;DR

BoxMed-RL tackles the lack of explainability in radiology report generation by introducing a two-phase training framework that combines radiologist-like chain-of-thought reasoning with spatially verifiable grounding to align findings with anatomical evidence, followed by a Lightweight Downstream Adapter to preserve fluency. It leverages a large vision-language backbone (Qwen2‑VL‑2B) and integrates Medical Concept Learning and Spatially Verifiable Reinforcement to mimic the radiologist workflow (Findings → Disease Category → Anatomy) and bound textual descriptions to bounding boxes using $IoU$‑based rewards. The method yields about a 7% gain in METEOR and ROUGE‑L and a 5% gain in large‑language‑model based metrics across public datasets, with ablations confirming the contributions of MCL and SVR. This work enhances clinical trust and spatial verifiability in automated radiology reports and points to future extensions in multi‑modal, 3D, and retrieval‑augmented radiology tasks, potentially enabling safer, more transparent AI in clinical workflows.

Abstract

Radiology report generation is critical for efficiency but current models lack the structured reasoning of experts, hindering clinical trust and explainability by failing to link visual findings to precise anatomical locations. This paper introduces BoxMed-RL, a groundbreaking unified training framework for generating spatially verifiable and explainable radiology reports. Built on a large vision-language model, BoxMed-RL revolutionizes report generation through two integrated phases: (1) In the Pretraining Phase, we refine the model via medical concept learning, using Chain-of-Thought supervision to internalize the radiologist-like workflow, followed by spatially verifiable reinforcement, which applies reinforcement learning to align medical findings with bounding boxes. (2) In the Downstream Adapter Phase, we freeze the pretrained weights and train a downstream adapter to ensure fluent and clinically credible reports. This framework precisely mimics radiologists' workflow, compelling the model to connect high-level medical concepts with definitive anatomical evidence. Extensive experiments on public datasets demonstrate that BoxMed-RL achieves an average 7% improvement in both METEOR and ROUGE-L metrics compared to state-of-the-art methods. An average 5% improvement in large language model-based metrics further underscores BoxMed-RL's robustness in generating high-quality radiology reports.

Figures (6)

• Figure 1: The motivation and challenges of our BoxMed-RL. (a) Clinical motivation: Radiologists interpret chest X-rays by combining spatial information with medical knowledge to identify findings, associate them with disease categories, and localize them in anatomical regions. (b) Challenges: Two major limitations are Workflow Mismatch and Textual-Visual Misalignment.
• Figure 2: Comparison of medical report generation architectures. (a) Traditional encoder-decoder framework. (b) Region-enhanced encoder-decoder models. (c) Prompt-driven large language models. (d) Our proposed BoxMed-RL.
• Figure 3: Overview of our BoxMed-RL framework, including 2 phases: (a) Pretraining Phase: The model first goes through MCL, enabling it to reason through structured medical concepts, mimicking radiologists’ diagnostic logic. Then it applies SVR, optimizing via reinforcement learning to align its concept reasoning with spatial regions in the image. (b) Downstream Adapter Phase: The model's weight is frozen and a downstream task adapter is refined to help the model generate fluent reports.
• Figure 4: Implement details of MCL module. (a) Illustration of our medical concept learning via CoT formulation. (b) presents the prompt template used to guide the model’s reasoning.
• Figure 5: Comprehensive analysis of each ablation module's performance on Green score 7 clinical evaluation criteria and CheXpert 14 categories classification. Bar charts display the detailed values for each category, while the radar chart on the right illustrates the overall performance of each ablated model. In the bar charts, the black line indicates the best performance in each category.