ResAgent: Entropy-based Prior Point Discovery and Visual Reasoning for Referring Expression Segmentation

TL;DR

This work tackles Referring Expression Segmentation (RES) by addressing two core weaknesses of existing MLLM-based approaches: coarse bounding boxes and text-based coordinate reasoning. It introduces ResAgent, a coarse-to-fine framework that combines Entropy-Based Point Discovery (EBD) with Vision-Based Reasoning (VBR) to select informative points within a predicted bounding box and validate them through visual grounding, minimizing reliance on textual coordinate tokens. EBD uses a structured spatial uncertainty field, modeled via a Bernoulli probability p(pt) and entropy H(pt), with a practical proxy realized by a superellipse spiral and dual-queue sampling. VBR leverages VQA-style prompts and marker-based visual reasoning to robustly verify candidate points, followed by a SAM-based decoder for mask generation, with LoRA-based adaptation to align with COCO annotation styles. Extensive experiments on RefCOCO, RefCOCO+, RefCOCOg, and ReasonSeg show state-of-the-art performance, demonstrating that minimal, information-rich prompts combined with visual grounding can achieve accurate, semantically grounded segmentation without heavy end-to-end tuning. The work also provides comprehensive analyses, ablations, and supplementary results across base models, highlighting robustness to bbox perturbations and offering insights into future enhancements such as reduced priors and more integrated end-to-end optimization.

Abstract

Referring Expression Segmentation (RES) is a core vision-language segmentation task that enables pixel-level understanding of targets via free-form linguistic expressions, supporting critical applications such as human-robot interaction and augmented reality. Despite the progress of Multimodal Large Language Model (MLLM)-based approaches, existing RES methods still suffer from two key limitations: first, the coarse bounding boxes from MLLMs lead to redundant or non-discriminative point prompts; second, the prevalent reliance on textual coordinate reasoning is unreliable, as it fails to distinguish targets from visually similar distractors. To address these issues, we propose \textbf{\model}, a novel RES framework integrating \textbf{E}ntropy-\textbf{B}ased Point \textbf{D}iscovery (\textbf{EBD}) and \textbf{V}ision-\textbf{B}ased \textbf{R}easoning (\textbf{VBR}). Specifically, EBD identifies high-information candidate points by modeling spatial uncertainty within coarse bounding boxes, treating point selection as an information maximization process. VBR verifies point correctness through joint visual-semantic alignment, abandoning text-only coordinate inference for more robust validation. Built on these components, \model implements a coarse-to-fine workflow: bounding box initialization, entropy-guided point discovery, vision-based validation, and mask decoding. Extensive evaluations on four benchmark datasets (RefCOCO, RefCOCO+, RefCOCOg, and ReasonSeg) demonstrate that \model achieves new state-of-the-art performance across all four benchmarks, highlighting its effectiveness in generating accurate and semantically grounded segmentation masks with minimal prompts.

Figures (7)

• Figure 1: Top: Accuracy (left) and confidence (right) of coordinate-based versus visual reasoning for point inference in a zero-shot setting. Bottom: Example queries highlighting the fundamental difference—textual coordinates lose spatial cues, while visual reasoning provides semantic anchoring.
• Figure 2: Overall framework of ResAgent. Starting with an MLLM-inferred bounding box as prior, the framework first uses Entropy-Based Point Discovery (EBD) to select high-value candidate points. Then ResAgent employs Vision-Based Reasoning (VBR) to validate points through visual markers and VQA queries, filtering noise. Early stopping is triggered when sufficient qualified points are collected, which are fed into the SAM decoder to generate accurate segmentation masks.
• Figure 3: Visualization of qualitative results. Each row shows one RES example, with subfigures: referring text, input image, baseline result, our ResAgent result, and GT (from left to right). Examples cover diverse challenging scenarios, and our ResAgent achieves more accurate segmentation consistent with GT.
• Figure 4: Visualization of the superellipse spiral sampling strategy across representative bounding box shapes. Each row corresponds to a different aspect ratio: elongated ($1\!:\!2$), moderately tall ($1\!:\!1.5$), and square ($1\!:\!1$) from top to bottom. Columns enumerate eight spiral configurations, defined by the combination of rotation direction (four clockwise followed by four counterclockwise) and terminal point (top, bottom, left, or right). hese variants demonstrate that the sampling density systematically concentrates in both center-proximal and boundary-proximal regions across all shapes and configurations. Note. Aspect ratios such as $1.5\!:\!1$ or $2\!:\!1$ are omitted because they are equivalent up to rotation (e.g., a $2\!:\!1$ box is simply a $1\!:\!2$ box rotated by $90^\circ$), and the sampling procedure is designed to be orientation-invariant in normalized coordinates.
• Figure 5: Comparison of confusion matrices for textual coordinate reasoning ('(x,y)') vs. VBR, under zeroshot validation on RefCOCO val split. VBR significantly increases true positives while reducing false positive and false negative errors.