IBISAgent: Reinforcing Pixel-Level Visual Reasoning in MLLMs for Universal Biomedical Object Referring and Segmentation

TL;DR

This work tackles the challenge of achieving pixel-level segmentation in medical MLLMs, where prior approaches rely on implicit tokens and single-turn reasoning. It introduces IBISAgent, an agentic MLLM that reframes segmentation as a multi-step, vision-centric decision process with interleaved reasoning and text-based clicks, invoking segmentation tools to refine masks while preserving the LLM's language capabilities. A large trajectory-annotated dataset is built, and a two-stage training pipeline (cold-start SFT followed by agentic RL with fine-grained rewards) is deployed, yielding state-of-the-art performance on multiple biomedical referring and segmentation benchmarks, including zero-shot. The results demonstrate robust pixel-level reasoning, flexible tool use, and scalable potential for holistic medical image analysis, with future directions including 3D extension and efficiency improvements.

Abstract

Recent research on medical MLLMs has gradually shifted its focus from image-level understanding to fine-grained, pixel-level comprehension. Although segmentation serves as the foundation for pixel-level understanding, existing approaches face two major challenges. First, they introduce implicit segmentation tokens and require simultaneous fine-tuning of both the MLLM and external pixel decoders, which increases the risk of catastrophic forgetting and limits generalization to out-of-domain scenarios. Second, most methods rely on single-pass reasoning and lack the capability to iteratively refine segmentation results, leading to suboptimal performance. To overcome these limitations, we propose a novel agentic MLLM, named IBISAgent, that reformulates segmentation as a vision-centric, multi-step decision-making process. IBISAgent enables MLLMs to generate interleaved reasoning and text-based click actions, invoke segmentation tools, and produce high-quality masks without architectural modifications. By iteratively performing multi-step visual reasoning on masked image features, IBISAgent naturally supports mask refinement and promotes the development of pixel-level visual reasoning capabilities. We further design a two-stage training framework consisting of cold-start supervised fine-tuning and agentic reinforcement learning with tailored, fine-grained rewards, enhancing the model's robustness in complex medical referring and reasoning segmentation tasks. Extensive experiments demonstrate that IBISAgent consistently outperforms both closed-source and open-source SOTA methods. All datasets, code, and trained models will be released publicly.

Figures (14)

• Figure 1: IBISAgent flexibly supports a wide range of fine-grained biomedical image understanding tasks, including referring and reasoning segmentation. It also handles a novel mask-refinement task that assists annotators in completing partially labeled masks.
• Figure 2: Overview of IBISAgent. (a) Overall architecture of the agent; (b) illustration of the cold-start SFT training process; and (c) illustration of the RL training process.
• Figure 3: Qualitative Analysis. We present the responses and segmentation outputs on a reasoning–segmentation example. Existing MLLMs exhibit incorrect reasoning and low-quality segmentation, highlighting their misaligned fine-grained vision–language understanding. In contrast, IBISAgent delivers substantially improved reasoning quality and segmentation performance.
• Figure 4: Modality distribution.(a) The RL corpus $\mathcal{D}_{rl}$ (training stage) contains $60,826$ samples and $564,385$ QAs across $8$ modalities. (b) The In-domain Test set $\mathcal{D}_{test}$ comprises $9,902$ samples and $156,289$ QAs covering the same 8 modalities. The dual-axis plots show the sample count (left axis) and total QA pairs (right axis) for each category.
• Figure 5: An illustrative example of the automated trajectory generation process for liver segmentation. The algorithm progressively refines the predicted mask through iterative interactions. For each iteration (e.g., Step 0), two visualization panels are presented: (1) The first image displays the current segmentation state, showing the predicted mask (green translucent overlay) generated by the current click (marked by a green star). (2) The second image illustrates the error analysis against the Ground Truth (delineated by a green outline). The differences are visualized as blue translucent regions for False Negatives (FN, under-segmentation) and red translucent regions for False Positives (FP, over-segmentation). The star in this panel indicates the calculated next action based on the largest error region: a blue star denotes a Positive Click (P) to correct under-segmentation, while a red star denotes a Negative Click (N) to correct over-segmentation.