Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examples

TL;DR

The paper presents a zero-annotation semantic segmentation pipeline that converts image-level prompts into per-category relevance maps using CLIP and transformer-based interpretability. By applying test-time augmentations and distractor calibration, these maps provide pseudo-supervision to downstream single-image segmentation methods, enabling high-quality masks without pixel-level labels. The approach achieves strong results on standard benchmarks, especially for rare categories, and demonstrates versatility by integrating with unsupervised and interactive segmentation. This work broadens practical segmentation capabilities to novel or underrepresented classes with minimal labeling effort.

Abstract

Semantic segmentation is a key computer vision task that has been actively researched for decades. In recent years, supervised methods have reached unprecedented accuracy, however they require many pixel-level annotations for every new class category which is very time-consuming and expensive. Additionally, the ability of current semantic segmentation networks to handle a large number of categories is limited. That means that images containing rare class categories are unlikely to be well segmented by current methods. In this paper we propose a novel approach for creating semantic segmentation masks for every object, without the need for training segmentation networks or seeing any segmentation masks. Our method takes as input the image-level labels of the class categories present in the image; they can be obtained automatically or manually. We utilize a vision-language embedding model (specifically CLIP) to create a rough segmentation map for each class, using model interpretability methods. We refine the maps using a test-time augmentation technique. The output of this stage provides pixel-level pseudo-labels, instead of the manual pixel-level labels required by supervised methods. Given the pseudo-labels, we utilize single-image segmentation techniques to obtain high-quality output segmentation masks. Our method is shown quantitatively and qualitatively to outperform methods that use a similar amount of supervision. Our results are particularly remarkable for images containing rare categories.

Figures (9)

• Figure 1: Results of our method on real-world images from rare class categories. As inputs, our algorithm receives an image along with text-prompts describing the classes that we want to segment. A language-vision model is distilled by generating a relevance map in relation to each prompt category. Further refinement is performed through test time augmentations. Next, the relevance maps are fed into a single image segmentation algorithm, which transforms the relevance maps into a high-quality segmentation.
• Figure 2: Segmentation Pipeline - A high-level description of our method. We receive an RGB image to process and user-suggested categories for segmentation. This assumes the prompt categories actually appear in the image. By using an interpretability method, we produce a relevance map with respect to each prompt category. Afterwards, we employ a segmentation algorithm (e.g. clustering or interactive segmentation algorithms) that is able to segment the image and leverage the pseudo labels induced by the relevance maps. Our final output is a segmentation map.
• Figure 3: Relevance Maps Generation - A sketch of the relevance map generation module, which can generate a relevance map for possibly any type of object. We utilize an interpretability method, a language-vision pre-trained model using TTA techniques, to obtain a refined relevance map.
• Figure 4: Crop Augmentation - Crops are selected from a grid overlaid on the image and relevance maps are calculated for each crop in relation to the prompt category. As crops overlap, each pixel's relevance value is averaged across different crops. In order to remove the noise and artifacts produced by the interpretability method, we subtracted the mean of the relevance maps of both the query category (e.g. Person, in green) and the distractor categories (Bird, Cat, Boat, and Bus, in red). Then, the probability of each class category is obtained using CLIP, which allows us to analyze only relevant labels, even if we do not know what the labels are. When our prompt category class probability is greater than a specified threshold (we used $\mathbb{P}(class) > 0.3$), we add the crop's relevance maps to an aggregated relevance map of the entire image. Finally, the aggregated relevance map is normalized to the range $[0,1]$, resulting in the final crop view.
• Figure 5: Obtaining clean relevance maps via Test Time Augmentation - Over a representative set of augmentations, a relevance map is created for each prompt category. By averaging all the relevance maps for each augmentation view, a more subtle map can be obtained for each category. While crop augmentation generally provides a finer map, other augmentations that work over the whole image usually produce a coarser map. Segmentation depends heavily on the map quality, which tends to be noisy.