Improving Medical Reasoning with Curriculum-Aware Reinforcement Learning

TL;DR

MedCCO introduces curriculum-guided reinforcement learning to jointly tackle close-ended and open-ended medical VQA within a single multimodal framework. It extends Group Relative Policy Optimization with verifiable rule-based rewards, deploying a two-stage training regimen that first instills domain-specific reasoning on close-ended data and then expands to open-ended tasks, aided by VQA data refinement. Across eight benchmarks, MedCCO achieves state-of-the-art performance in-domain and robust generalization out-of-domain, including cross-modal transfer on SLAKE, while ablations show curriculum learning and data refinement are key to stability and performance. This approach offers a scalable, data-efficient path to robust, interpretable medical reasoning without reliance on handcrafted CoT explanations, potentially easing deployment in clinical settings.

Abstract

Recent advances in reinforcement learning with verifiable, rule-based rewards have greatly enhanced the reasoning capabilities and out-of-distribution generalization of VLMs/LLMs, obviating the need for manually crafted reasoning chains. Despite these promising developments in the general domain, their translation to medical imaging remains limited. Current medical reinforcement fine-tuning (RFT) methods predominantly focus on close-ended VQA, thereby restricting the model's ability to engage in world knowledge retrieval and flexible task adaptation. More critically, these methods fall short of addressing the critical clinical demand for open-ended, reasoning-intensive decision-making. To bridge this gap, we introduce \textbf{MedCCO}, the first multimodal reinforcement learning framework tailored for medical VQA that unifies close-ended and open-ended data within a curriculum-driven RFT paradigm. Specifically, MedCCO is initially fine-tuned on a diverse set of close-ended medical VQA tasks to establish domain-grounded reasoning capabilities, and is then progressively adapted to open-ended tasks to foster deeper knowledge enhancement and clinical interpretability. We validate MedCCO across eight challenging medical VQA benchmarks, spanning both close-ended and open-ended settings. Experimental results show that MedCCO consistently enhances performance and generalization, achieving a 11.4\% accuracy gain across three in-domain tasks, and a 5.7\% improvement on five out-of-domain benchmarks. These findings highlight the promise of curriculum-guided RL in advancing robust, clinically-relevant reasoning in medical multimodal language models.

Figures (5)

• Figure 1: Overview of our MedCCO. MedCCO is first fine-tuned with GRPO on close-ended medical VQA to establish fundamental reasoning capabilities, then adapted to open-ended tasks to enable more advanced reasoning and knowledge retrieval. Prior to open-ended GRPO, we refine VQA data consistency to improve training efficiency.
• Figure 2: Cross-modal performance on SLAKE liu2021slake, with each model trained on a single modality and evaluated across all modalities for in- and cross-modal comparison.
• Figure 3: Training curves and overall performance with and without VQA refinement. Incorporating VQA refinement leads to more stable training and improved generalization.
• Figure 4: Qualitative results of open-ended and close-ended VQA reasoning. (a) and (b) show open-ended VQA without an option list, while (c) illustrates a close-ended VQA with an option list.
• Figure 5: Prompt for refining open-ended VQA consistency.