Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation

TL;DR

Segment Anyword introduces a training-free prompt-learning framework for open-set language grounded segmentation that leverages token-level cross-attention from a frozen diffusion model to generate segmentation prompts. By updating only the textual embeddings at test time and refining initial cross-attention prompts with linguistic-guided regularization, the method achieves robust, noise-tolerant masks without extensive training. The approach integrates a diffusion-based localization prior with a SAM post-processor and demonstrates state-of-the-art performance among training-free methods across GranDf, gRefCOCO, and PC-59, while showing strong generalization to open-set and reference segmentation scenarios. The results highlight the practical impact of combining diffusion-model attention, test-time prompt learning, and linguistic structure to enhance open-vocabulary segmentation in real-world settings.

Abstract

Open-set image segmentation poses a significant challenge because existing methods often demand extensive training or fine-tuning and generally struggle to segment unified objects consistently across diverse text reference expressions. Motivated by this, we propose Segment Anyword, a novel training-free visual concept prompt learning approach for open-set language grounded segmentation that relies on token-level cross-attention maps from a frozen diffusion model to produce segmentation surrogates or mask prompts, which are then refined into targeted object masks. Initial prompts typically lack coherence and consistency as the complexity of the image-text increases, resulting in suboptimal mask fragments. To tackle this issue, we further introduce a novel linguistic-guided visual prompt regularization that binds and clusters visual prompts based on sentence dependency and syntactic structural information, enabling the extraction of robust, noise-tolerant mask prompts, and significant improvements in segmentation accuracy. The proposed approach is effective, generalizes across different open-set segmentation tasks, and achieves state-of-the-art results of 52.5 (+6.8 relative) mIoU on Pascal Context 59, 67.73 (+25.73 relative) cIoU on gRefCOCO, and 67.4 (+1.1 relative to fine-tuned methods) mIoU on GranDf, which is the most complex open-set grounded segmentation task in the field.

Figures (24)

• Figure 1: Key comparison of multi-modal open-set image segmentation architectures. (a) CLIP based methods wang2022crisxu2023bridging (b) MLLM based methods rasheed2024glammlai2024lisazhang2024omg. (c) Previous prompt learning based methods chen2025sam4mllmlin2024training (d) Our Segment Anyword's architecture, which possesses a simple design for effective arbitrary syntax level grounded segmentation capability with minimal visual prompt optimizing efforts. Project page, code, and data are available at https://zhihualiued.github.io/segment_anyword
• Figure 2: We propose Segment Anyword for Open-Set Grounded Segmentation. Segment Anyword is multi-modal promptable image segmentor solely built on the semantic prior knowledge extracted from a frozen diffusion model. Segment Anyword demonstrate superior performance across multiple multi-modal segmentation task, including 1) reference image segmentation (refCOCO, gRefCOCO), 2) complex grounding segmentation (GranDf) and 3) OOD Medical Image Segmentation. Best viewed in colors and zoomed-in.
• Figure 3: Motivational Study on the multi-object reference segmentation ReLA liu2023gres. We categorize data points into easy (green), medium (blue), and hard (red) samples to achieve accurate and stable segmentation, based on the retained IoU mean and standard deviation, which are calculated across the caption dimension. The IoU for each image-text pair, compared to the ground truth, is shown in brackets. Our study validates the presence of segmentation variability, highlighting the challenges in generating accurate and stable segmentation masks with free-form text reference descriptions.
• Figure 4: Pipeline overview. Segment Anyword leverage the inversed scalability of a frozen text-to-image denosing diffusion model $\epsilon_{\theta}$ for training-free open-set language grounded segmentation. First, Segment Anyword regards the segmentation reference expression (top-left) as image generation text condition and reconstruct the input image $x_0$. Within image reconstruction process, Segment Anyword only update the textual embedding $V = [v^\star,\dots,v^{\And}]$ of visual concepts (coloured texts) while the rest of network parameters ($c_{\phi}$, $\epsilon_{\theta}$) remain frozen. At test time, the averaged cross-attention maps are collected through an diffusion process of $Z^\star$ with optimized $V$. We further introduce novel linguistic-guided visual prompt regularization to bind and cluster mask prompts for mining noise-tolerant prompts and improve downstream segmentation accuracy by integrating sentence denpendency and syntax structual information directly.
• Figure 5: Illustration of the averaged cross-attention map and attention mask. By leveraging a frozen denoising text-to-image diffusion network, the word-level cross-attention map offers a location prior. In contrast, the hard mask produces a sub-optimal segmentation result, lacking fine-grained details such as object shape, posture, and boundaries.