RSAgent: Learning to Reason and Act for Text-Guided Segmentation via Multi-Turn Tool Invocations

TL;DR

RSAgent tackles text-guided segmentation by framing it as an interactive, episodic decision process in which a multimodal LLM alternates between reasoning and multi-turn tool invocations to query a segmentation toolbox and iteratively refine masks. A dedicated data pipeline generates high-quality multi-turn trajectories for cold-start supervised fine-tuning, followed by agentic reinforcement learning with fine-grained rewards that reward final mask quality and step-wise improvements. The two-stage training, combined with a modular tool environment, enables robust cross-modal reasoning and targeted refinement, achieving state-of-the-art results on RES and ReasonSeg benchmarks (e.g., 66.5% gIoU on ReasonSeg and 81.5% cIoU on RefCOCOg). This approach demonstrates a strong, interactive paradigm for text-guided pixel-level understanding, with potential impact on interactive perception and embodied AI tasks that require precise localization under open vocabulary.

Abstract

Text-guided object segmentation requires both cross-modal reasoning and pixel grounding abilities. Most recent methods treat text-guided segmentation as one-shot grounding, where the model predicts pixel prompts in a single forward pass to drive an external segmentor, which limits verification, refocusing and refinement when initial localization is wrong. To address this limitation, we propose RSAgent, an agentic Multimodal Large Language Model (MLLM) which interleaves reasoning and action for segmentation via multi-turn tool invocations. RSAgent queries a segmentation toolbox, observes visual feedback, and revises its spatial hypothesis using historical observations to re-localize targets and iteratively refine masks. We further build a data pipeline to synthesize multi-turn reasoning segmentation trajectories, and train RSAgent with a two-stage framework: cold-start supervised fine-tuning followed by agentic reinforcement learning with fine-grained, task-specific rewards. Extensive experiments show that RSAgent achieves a zero-shot performance of 66.5% gIoU on ReasonSeg test, improving over Seg-Zero-7B by 9%, and reaches 81.5% cIoU on RefCOCOg, demonstrating state-of-the-art performance on both in-domain and out-of-domain benchmarks.

Figures (5)

• Figure 1: Performance on text-guided object segmentation benchmarks. RSAgent achieves state-of-the-art on both RES and ReasonSeg benchmarks.
• Figure 2: Comparing to LISA's direct segmentation and Seg-Zero's single forward pass of thinking and segmentation, RSAgent operates by iteratively proposing and updating spatial prompts and invoking external visual tools to iteratively refine the final mask.
• Figure 3: Overview of RSAgent. Given the original image and problem, the agent interacts with an external visual toolbox over multiple rounds, incrementally gathers visual evidence, refines candidate masks, and eventually committs to a final prediction. RSAgent first embraces cold-start SFT to get accustomed to reasoning and multi-turn tool invocating operations via the cold-start data generated by our data pipeline, then gets optimized by RL with fine-grained rewards.
• Figure 4: The multi-turn reasoning segmentation data pipeline, including problem generation, trajectory synthesis and data filtering.
• Figure 5: Effect of the maximum number of tool-invocation turns during training. Metrics are cIoU (%).