MEDVISTAGYM: A Scalable Training Environment for Thinking with Medical Images via Tool-Integrated Reinforcement Learning

TL;DR

MedVistaGym introduces a scalable, tool-integrated training environment for medical visual reasoning and presents MedVista-R1, an agent trained with a two-stage pipeline (cold-start supervised bootstrapping followed by online reinforcement learning) to interleave reasoning with tool invocations. Formulated as a POMDP with explicit think and tool steps, the framework allows dynamic tool use and evidence-grounded decisions across six medical VQA benchmarks, including both in-domain and out-of-domain tasks. Empirical results show robust gains over comparably sized baselines, with 8B MedVista-R1 achieving 19.1–24.2% improvements when using tool-enabled RL, and 8B models delivering strong parameter efficiency. The work demonstrates that structured agentic training, diverse tool sets, and scalable asynchronous execution are crucial to unlocking effective tool-integrated reasoning for medical image analysis, while acknowledging computational costs and tool-output noise as ongoing challenges.

Abstract

Vision language models (VLMs) achieve strong performance on general image understanding but struggle to think with medical images, especially when performing multi-step reasoning through iterative visual interaction. Medical VLMs often rely on static visual embeddings and single-pass inference, preventing models from re-examining, verifying, or refining visual evidence during reasoning. While tool-integrated reasoning offers a promising path forward, open-source VLMs lack the training infrastructure to learn effective tool selection, invocation, and coordination in multi-modal medical reasoning. We introduce MedVistaGym, a scalable and interactive training environment that incentivizes tool-integrated visual reasoning for medical image analysis. MedVistaGym equips VLMs to determine when and which tools to invoke, localize task-relevant image regions, and integrate single or multiple sub-image evidence into interleaved multimodal reasoning within a unified, executable interface for agentic training. Using MedVistaGym, we train MedVistaGym-R1 to interleave tool use with agentic reasoning through trajectory sampling and end-to-end reinforcement learning. Across six medical VQA benchmarks, MedVistaGym-R1-8B exceeds comparably sized tool-augmented baselines by 19.10% to 24.21%, demonstrating that structured agentic training--not tool access alone--unlocks effective tool-integrated reasoning for medical image analysis.

Figures (15)

• Figure 1: Overview of MedVistaGym, which contains a comprehensive suite of reasoning-intensive medical image analysis tasks and tools in an interactive execution environment, scaling visual-centric tool-integrated agentic reinforcement learning for VLM agents.
• Figure 2: Ablation study on training configurations.
• Figure 3: Error analysis.
• Figure 4: Effect of thinking trajectory length.
• Figure 5: Prompt template for trajectory quality evaluation.