Med-VRAgent: A Framework for Medical Visual Reasoning-Enhanced Agents

TL;DR

The paper tackles hallucination and localization challenges in medical visual reasoning by introducing Med-VRAgent, a multimodal agent that combines Visual Guidance, ROI-focused perception, a Teacher-Student-Assessor loop, Retrieval-Augmented Reflection, and Monte Carlo Tree Search. It uses PPO-based fine-tuning on collected reasoning trajectories to align the model with medical reasoning tasks. Empirical results across IU-Xray, MIMIC-CXR, VQA-RAD, and GMAI-MMbench demonstrate state-of-the-art performance on both report generation and visual question answering, outperforming strong baselines and retrieval-based methods. The approach contributes structured reasoning trajectories, improved ROI grounding, and external knowledge grounding, with potential impact on clinical reliability and applicability, while acknowledging computational costs and transferability considerations for real-world deployment.

Abstract

Visual Language Models (VLMs) achieve promising results in medical reasoning but struggle with hallucinations, vague descriptions, inconsistent logic and poor localization. To address this, we propose a agent framework named Medical Visual Reasoning Agent (\textbf{Med-VRAgent}). The approach is based on Visual Guidance and Self-Reward paradigms and Monte Carlo Tree Search (MCTS). By combining the Visual Guidance with tree search, Med-VRAgent improves the medical visual reasoning capabilities of VLMs. We use the trajectories collected by Med-VRAgent as feedback to further improve the performance by fine-tuning the VLMs with the proximal policy optimization (PPO) objective. Experiments on multiple medical VQA benchmarks demonstrate that our method outperforms existing approaches.

Figures (7)

• Figure 1: Top: A student struggles, feeling confused and making mistakes. Bottom: With guidance, the student overcomes the confusion and successfully completes the task.
• Figure 2: Overview of the Med-VRAgent framework. The system uses MCTS to generate solutions $\mathcal{S}_{ij}$ based on Regions of Interest $\mathcal{ROI}_{ij}$, visual guides $\mathcal{G}_{ij}$, rewards $\mathcal{R}_{ij}$, and external knowledge $\mathcal{K}_{ij}$.$\mathcal{S}^*_{ij}$ is the solution after reflection.
• Figure 3: Ablation Experiment 1 Results (accuracy; %) for DeepSeek-VL-7B with Med-VRAgent on dataset GMAI-MMBench. Noall means removing all components, NoV means removing visual extraction, NOA means removing Assessor, and NoT means removing Teacher.
• Figure 4: Ablation Experiment 2 Results (accuracy; %) for DeepSeek-VL-7B with Med-VRAgent on dataset GMAI-MMBench. Best is an adaptive exploration strategy, the average width and depth are 1.74 and 2.23 (red line), and other combinations are fixed width and depth.
• Figure 5: Med-VRAgent Medical Report Generation Case Study