RESAnything: Attribute Prompting for Arbitrary Referring Segmentation

TL;DR

RESAnything introduces a training-free, open-vocabulary approach to arbitrary referring segmentation by combining attribute prompting with a two-stage, multi-metric mask selection framework. The method generates rich reference and candidate texts via LLMs grounded on SAM proposals, and uses text-to-text and text-to-image comparisons along with CLIP scores to robustly select the correct mask, including cases involving implicit, part-level, and materials- or function-based references. A new ABO-Image-ARES dataset with ~3K carefully annotated expressions evaluates part-level and arbitrary RES, where RESAnything achieves state-of-the-art zero-shot performance on vanilla RES and superior reasoning segmentation results on ReasonSeg and ABO. The work demonstrates strong generalization, efficiency, and practical impact for real-world tasks requiring fine-grained, open-ended referential understanding without task-specific fine-tuning.

Abstract

We present an open-vocabulary and zero-shot method for arbitrary referring expression segmentation (RES), targeting input expressions that are more general than what prior works were designed to handle. Specifically, our inputs encompass both object- and part-level labels as well as implicit references pointing to properties or qualities of object/part function, design, style, material, etc. Our model, coined RESAnything, leverages Chain-of-Thoughts (CoT) reasoning, where the key idea is attribute prompting. We generate detailed descriptions of object/part attributes including shape, color, and location for potential segment proposals through systematic prompting of a large language model (LLM), where the proposals are produced by a foundational image segmentation model. Our approach encourages deep reasoning about object or part attributes related to function, style, design, etc., enabling the system to handle implicit queries without any part annotations for training or fine-tuning. As the first zero-shot and LLM-based RES method, RESAnything achieves clearly superior performance among zero-shot methods on traditional RES benchmarks and significantly outperforms existing methods on challenging scenarios involving implicit queries and complex part-level relations. Finally, we contribute a new benchmark dataset to offer ~3K carefully curated RES instances to assess part-level, arbitrary RES solutions.

Figures (31)

• Figure 1: Open-vocabulary and zero-shot referring expression segmentation with RESAnything. Our method produces accurate object or part masks from general- and free-form text expressions including, from left to right: object or part semantic label, material/style properties, function/design descriptions, or logos and packaging labels in textual or other graphical in an image. For visualization purposes, we overlay segmentation regions with red color in each example.
• Figure 2: Overview of RESAnything: a two-stage framework for zero-shot arbitrary RES. The attribute prompting stage generates reference and candidate texts from input image and referring expression using SAM-generated proposals and an MLLM. The mask proposal selection stage leverages MLLM and CLIP to evaluate both candidates and proposals and produce the final response.
• Figure 3: Attribute prompting using SAM and MLLM. Given the input image and referring expression, this stage produces two groups of predictions. The first output, a reference text $T^\text{ref}$, is generated from an MLLM with the text prompt $Q^\text{ref}$. It describes the visual attributes (e.g., color, shape, location) of the target region ("mesh backrest" in this example). The second group is a set of candidate texts $T_i^\text{can}$, generated by an MLLM with the text prompt $Q^\text{can}$ and visual prompts derived from segmentation mask proposals. These texts describe the attributes of their corresponding segmentation region proposals, visualized with the same border color.
• Figure 4: Example of different visual prompts $V_i$ generated from a segmentation proposal $m_i$.
• Figure 5: Multi-metric mask proposal selection using MLLM and CLIP. To select the final mask from mask proposals generated by SAM, we introduce four metrics computed across different modalities and models to evaluate the similarity between input expression $E$ and the mask proposals. Specifically, the text-to-text MLLM-based binary decision $d^\text{t2t}$ and CLIP score $s^\text{t2t}$ match reference text to candidate texts. The text-to-image MLLM-based binary decision $d^\text{t2i}$ and CLIP score $s^\text{t2i}$ match reference text to masked visual prompts.