Image Segmentation Using Text and Image Prompts

TL;DR

This work introduces CLIPSeg, a unified, prompt-driven image segmentation model built on a frozen CLIP backbone with a lightweight transformer decoder. It supports segmentation conditioned by either text or visual prompts, enabling three tasks—referring expression segmentation, generalized zero-shot segmentation, and one-shot segmentation—within a single framework. A key innovation is PhraseCut+ (PC+), which adds visual supports and negatives to train a model capable of image-text interpolation and robust generalization, including to affordance-based prompts. Extensive experiments across multiple benchmarks demonstrate competitive performance and strong generalization, without task-specific retraining, highlighting the practicality of prompt-based segmentation in real-world settings such as robotics. The work also analyzes prompt engineering strategies, showing that visual prompting and CLIP’s cross-modal embeddings are crucial for robust, open-vocabulary segmentation.

Abstract

Image segmentation is usually addressed by training a model for a fixed set of object classes. Incorporating additional classes or more complex queries later is expensive as it requires re-training the model on a dataset that encompasses these expressions. Here we propose a system that can generate image segmentations based on arbitrary prompts at test time. A prompt can be either a text or an image. This approach enables us to create a unified model (trained once) for three common segmentation tasks, which come with distinct challenges: referring expression segmentation, zero-shot segmentation and one-shot segmentation. We build upon the CLIP model as a backbone which we extend with a transformer-based decoder that enables dense prediction. After training on an extended version of the PhraseCut dataset, our system generates a binary segmentation map for an image based on a free-text prompt or on an additional image expressing the query. We analyze different variants of the latter image-based prompts in detail. This novel hybrid input allows for dynamic adaptation not only to the three segmentation tasks mentioned above, but to any binary segmentation task where a text or image query can be formulated. Finally, we find our system to adapt well to generalized queries involving affordances or properties. Code is available at https://eckerlab.org/code/clipseg.

Figures (8)

• Figure 1: Our key idea is to use CLIP to build a flexible zero/one-shot segmentation system that addresses multiple tasks at once.
• Figure 2: Architecture of CLIPSeg: We extend a frozen CLIP model (red and blue) with a transformer that segments the query image based on either a support image or a support prompt. $N$ CLIP activations are extracted after blocks defined by $\mathcal{S}$. The segmentation transformer and the projections (both green) are trained on PhraseCut or PhraseCut+.
• Figure 3: Different forms of combining an image with the associated object mask to build a visual prompt have a strong effect on CLIP predictions (bar charts). We use the difference in the probability of the target object (orange) in the original image (left column) and the masking methods for our systematic analysis.
• Figure 4: Qualitative predictions of CLIPSeg (PC+) for various prompts, darkness indicates prediction strength. The generalized prompts (left) deviate from the PhraseCut prompts as they involve action-related properties or new object names.
• Figure 5: Image classification performance of CLIP over different image sizes.