MedSAM-Agent: Empowering Interactive Medical Image Segmentation with Multi-turn Agentic Reinforcement Learning

TL;DR

This work tackles the generalization gap in medical image segmentation by introducing MedSAM-Agent, an autonomous multi-turn agent that learns to interact with segmentation tools through expert-curated trajectories. It combines a hybrid prompting strategy (Box-to-Point and Sequential-Click) with a two-stage training pipeline (SFT followed by RLVR) and a granular, clinically grounded reward design to optimize both accuracy and interaction efficiency. The approach demonstrates state-of-the-art performance across 6 modalities and 21 datasets, and exhibits zero-shot generalization to different segmentation backends, indicating robust, tool-agnostic strategic understanding. Overall, MedSAM-Agent offers a scalable path toward autonomous, efficient, and clinically faithful interactive segmentation capable of integrating into diverse clinical workflows.

Abstract

Medical image segmentation is evolving from task-specific models toward generalizable frameworks. Recent research leverages Multi-modal Large Language Models (MLLMs) as autonomous agents, employing reinforcement learning with verifiable reward (RLVR) to orchestrate specialized tools like the Segment Anything Model (SAM). However, these approaches often rely on single-turn, rigid interaction strategies and lack process-level supervision during training, which hinders their ability to fully exploit the dynamic potential of interactive tools and leads to redundant actions. To bridge this gap, we propose MedSAM-Agent, a framework that reformulates interactive segmentation as a multi-step autonomous decision-making process. First, we introduce a hybrid prompting strategy for expert-curated trajectory generation, enabling the model to internalize human-like decision heuristics and adaptive refinement strategies. Furthermore, we develop a two-stage training pipeline that integrates multi-turn, end-to-end outcome verification with a clinical-fidelity process reward design to promote interaction parsimony and decision efficiency. Extensive experiments across 6 medical modalities and 21 datasets demonstrate that MedSAM-Agent achieves state-of-the-art performance, effectively unifying autonomous medical reasoning with robust, iterative optimization. Code is available \href{https://github.com/CUHK-AIM-Group/MedSAM-Agent}{here}.

Figures (4)

• Figure 1: Comparison of medical image segmentation paradigms. (a) SAM-based models (e.g., SAM, MedSAM) require continuous manual prompting via points or bounding boxes. (b) MLLM-driven models (e.g., LISA, UniBioMed) employs MLLM with specialized seg decoders and <seg> tokens. (c) Ours MedSAM-Agent functions as an autonomous visual agent that performs multi-turn refinement through iterative feedback and tool interaction, emulating the professional decision-making process.
• Figure 2: Overview of MedSAM-Agent. We develop a hybrid prompting strategy for expert-curated trajectory generation that transforms image-label pairs into high-quality interaction sequences via simulated clicks and IoU-based filtering. Then these trajectories support a two-stage training pipeline, stage-1 SFT cold-start for initial capability and stage-2 RL optimized by a fine-grained reward design. MedSAM-Agent can autonomously select between box and point tools and execute the "stop" action once the refinement is complete.
• Figure 3: Analysis of multi-turn interaction. The orange and purple lines represent the performance of the static Single-Turn Box and Single-Turn Point prompts, respectively, where inputs are derived from ground-truth bounding boxes and centroids. The blue line plots the Mean IoU across successive interaction turns. The bar charts illustrate the distribution of sample outcomes at each turn, where green, red, and grey segments denote the proportion of samples exhibiting improved, declined, or unchanged IoU. The segmentation tool is IMISNet IMISNet.
• Figure 4: Case study. Yellow boxes indicate bounding box prompts, yellow points represent positive clicks, and red points denote negative clicks.