Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DR$^2$Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models

Yulin He, Wei Chen, Zhikang Jian, Tianhang Guo, Wenjuan Zhou, Minglong Li

TL;DR

Reasoning segmentation often suffers from overlong, unfocused reasoning that can impede object localization. DR$^2$Seg introduces a decomposed two-stage rollout and self-reward framework to separate multimodal reasoning from referring segmentation, using a first pass to generate an explicit description and a second pass to verify it. The approach defines a total reward that combines a base signal with description fidelity and a length-based penalty, optimized via GRPO without requiring extra MLLMs. Across zero-shot and few-shot settings on ReasonSeg and related tasks, DR$^2$Seg yields consistent gains in segmentation accuracy while dramatically reducing thinking tokens, highlighting a scalable path for efficient multimodal reasoning in LLM-based vision systems.

Abstract

Reasoning segmentation is an emerging vision-language task that requires reasoning over intricate text queries to precisely segment objects. However, existing methods typically suffer from overthinking, generating verbose reasoning chains that interfere with object localization in multimodal large language models (MLLMs). To address this issue, we propose DR$^2$Seg, a self-rewarding framework that improves both reasoning efficiency and segmentation accuracy without requiring extra thinking supervision. DR$^2$Seg employs a two-stage rollout strategy that decomposes reasoning segmentation into multimodal reasoning and referring segmentation. In the first stage, the model generates a self-contained description that explicitly specifies the target object. In the second stage, this description replaces the original complex query to verify its self-containment. Based on this design, two self-rewards are introduced to strengthen goal-oriented reasoning and suppress redundant thinking. Extensive experiments across MLLMs of varying scales and segmentation models demonstrate that DR$^2$Seg consistently improves reasoning efficiency and overall segmentation performance.

DR$^2$Seg: Decomposed Two-Stage Rollouts for Efficient Reasoning Segmentation in Multimodal Large Language Models

TL;DR

Reasoning segmentation often suffers from overlong, unfocused reasoning that can impede object localization. DRSeg introduces a decomposed two-stage rollout and self-reward framework to separate multimodal reasoning from referring segmentation, using a first pass to generate an explicit description and a second pass to verify it. The approach defines a total reward that combines a base signal with description fidelity and a length-based penalty, optimized via GRPO without requiring extra MLLMs. Across zero-shot and few-shot settings on ReasonSeg and related tasks, DRSeg yields consistent gains in segmentation accuracy while dramatically reducing thinking tokens, highlighting a scalable path for efficient multimodal reasoning in LLM-based vision systems.

Abstract

Reasoning segmentation is an emerging vision-language task that requires reasoning over intricate text queries to precisely segment objects. However, existing methods typically suffer from overthinking, generating verbose reasoning chains that interfere with object localization in multimodal large language models (MLLMs). To address this issue, we propose DRSeg, a self-rewarding framework that improves both reasoning efficiency and segmentation accuracy without requiring extra thinking supervision. DRSeg employs a two-stage rollout strategy that decomposes reasoning segmentation into multimodal reasoning and referring segmentation. In the first stage, the model generates a self-contained description that explicitly specifies the target object. In the second stage, this description replaces the original complex query to verify its self-containment. Based on this design, two self-rewards are introduced to strengthen goal-oriented reasoning and suppress redundant thinking. Extensive experiments across MLLMs of varying scales and segmentation models demonstrate that DRSeg consistently improves reasoning efficiency and overall segmentation performance.
Paper Structure (14 sections, 11 equations, 4 figures, 7 tables)

This paper contains 14 sections, 11 equations, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reasoning Segmentation
  4. Self-Rewarding Reinforcement Learning
  5. Methodology
  6. Problem Definition
  7. Overview
  8. DR$^2$Seg: A Pure Self-Reward Framework
  9. Theoretical Analysis
  10. Experiments
  11. Experimental Settings
  12. Main Results
  13. Diagnostic Experiments
  14. Conclusions

Figures (4)

  • Figure 1: Motivation of DR$^2$Seg. Verbose reasoning can mislead MLLMs to localize false regions (e.g., face or whiskers) instead of the true target (nose). DR$^2$Seg mitigates such reasoning and localization errors, achieving efficient reasoning ($\sim$3× shorter) and accurate segmentation on RefCOCO (RCO) and ReasonSeg (RS).
  • Figure 2: Overview of DR$^2$Seg. (a) DR$^2$Seg performs a two-stage rollout. In this first pass, the model takes an image-query pair and produces a structured output comprising a CoT, a description, and an answer. In the second pass, the model is re-prompted with the image and the generated description, replacing the original query. (b) DR$^2$Seg adopts a self-reward mechanism to optimize the MLLM, enabling more efficient reasoning and accurate segmentation.
  • Figure 3: Qualitative comparisons between VisionReasoner and our DR$^2$Seg. The representative samples are selected from simple single-object to complex multi-object scenarios.
  • Figure 4: Effect of the Two-Stage Rollout Strategy. We analyze the evolution of answer entropy, thinking token count, and accuracy during training, where answer entropy reflects the model’s output uncertainty.