LVMed-R2: Perception and Reflection-driven Complex Reasoning for Medical Report Generation

TL;DR

This work addresses the lack of complex reasoning and self-verification in large vision-language models for medical report generation by introducing LVMed-R2, a fine-tuning strategy that combines a perception-tree guided complex reasoning pipeline with a reflection mechanism. The method injects medical knowledge and perception enhancements, then applies a self-verification loop to refine outputs. Built on IU-Xray and MIMIC-CXR, LVMed-R2 demonstrates 8–12% gains in natural language generation metrics and 7–10% gains in clinical-efficacy metrics over strong baselines, indicating improved diagnostic reasoning and reduced perceptual errors. The approach paves the way for more reliable, information-rich radiology reports and motivates future work on extending reasoning-augmented training to broader medical tasks and benchmarking.

Abstract

Large vision-language models (LVMs) hold a great promise for automating medical report generation, potentially reducing the burden of manual reporting. State-of-the-art (SOTA) research fine-tunes general LVMs with medical data to align radiology images to corresponding medical reports. However, there are two key factors that limit these LVM's performance. Firstly, LVMs lack complex reasoning capability that leads to logical inconsistencies and potential diagnostic errors in generated reports. Secondly, LVMs lack reflection mechanism that leads to an inability to discover errors in the thinking process. To address these gaps, we propose LVMed-R2, a new fine-tuning strategy that introduces complex reasoning and reflection mechanisms for LVMs to enhance medical report generation. To the best of our knowledge, this is the first work to introduce complex reasoning to the medical report generation (MRG) task. Our proposed complex reasoning contains medical knowledge injection and perception-enhancing modules which improve the accuracy of LVMs diagnosis, coupled with a perception tree to provide guidance to limit the perception range. Further, the reflection mechanism forces self-verification for outputs to correct for potential errors. We experimented by fine-tuning LVMs with our proposed LVMed-R2 strategy, using IU-Xray and MIMIC-CXR datasets. Our results, measured on natural language generation (NLG) metrics and clinical efficacy (CE) metrics, demonstrate that LVMs fine-tuned with the proposed reflection mechanism possess the ability to correct outputs and complex reasoning effectively and improve LVMs performance for MRG.

Figures (3)

• Figure 1: Demonstration of constructing complex reasoning with perception-enhanced and medical knowledge injection, and activating reflection mechanism. Left: The construction process of complex reasoning and perception tree. Right: Introducing reflection mechanism based on complex reasoning.
• Figure 2: Three typical cases generated by LVMs which were fine-tuned with LVM-R strategy. Case A demonstrates the style refinement of generated report. Case B demonstrates the accurate diagnosis utilizing medical knowledge. Case C demonstrates the perception enhancing module to eliminate wrong recognition.
• Figure 3: A patient case demonstrating the reflection mechanism on perception and report generation modules.