A Semantic Space is Worth 256 Language Descriptions: Make Stronger Segmentation Models with Descriptive Properties

TL;DR

This work introduces ProLab, a semantic segmentation framework that replaces traditional one-hot category labels with a property-level space derived from Large Language Models. Descriptions of categories are embedded and clustered into an interpretable set of properties, which supervision segmentation via multi-label property logits and cosine-based similarities to recover category predictions. Across five classic benchmarks, ProLab yields stronger performance and better scalability, and demonstrates generalization to out-of-domain or unknown categories using in-domain descriptive properties. The approach also shows adaptability to other segmentation architectures, offering a path toward more human-aligned, open-vocabulary segmentation with improved interpretability and robustness.

Abstract

This paper introduces ProLab, a novel approach using property-level label space for creating strong interpretable segmentation models. Instead of relying solely on category-specific annotations, ProLab uses descriptive properties grounded in common sense knowledge for supervising segmentation models. It is based on two core designs. First, we employ Large Language Models (LLMs) and carefully crafted prompts to generate descriptions of all involved categories that carry meaningful common sense knowledge and follow a structured format. Second, we introduce a description embedding model preserving semantic correlation across descriptions and then cluster them into a set of descriptive properties (e.g., 256) using K-Means. These properties are based on interpretable common sense knowledge consistent with theories of human recognition. We empirically show that our approach makes segmentation models perform stronger on five classic benchmarks (e.g., ADE20K, COCO-Stuff, Pascal Context, Cityscapes, and BDD). Our method also shows better scalability with extended training steps than category-level supervision. Our interpretable segmentation framework also emerges with the generalization ability to segment out-of-domain or unknown categories using only in-domain descriptive properties. Code is available at https://github.com/lambert-x/ProLab.

Figures (8)

• Figure 1: Advantages of ProLab. Compared to classic category-level label space, ProLab improves segmentation models in three aspects: (a) stronger performance on classic segmentation benchmarks (b) better scalability with extended training steps (c) ability to segment by descriptive properties, which could generalize to out-of-domain categories or even unknown categories.
• Figure 2: Build a semantic space of descriptive properties. ProLab firstly employs a Large Language Model to extract common sense knowledge pertinent to all involved categories, utilizing crafted prompts to ensure a structured format. Subsequently, a description embedding model is used to encode these descriptions, preserving semantic correlations. Finally, the description embeddings are grouped into a series of unique descriptive properties through K-Means clustering.
• Figure 3: Prompt for descriptions. This crafted-prompt ensures precise guidance for Large Language Models in retrieving consistent and reliable property descriptions.
• Figure 4: Supervise and classify with properties. Left: the training procedure where descriptive properties are used for model supervision. Right: the inference procedure of categorizing items within the original category-level label space.
• Figure 5: Segmentation with interpretable properties. Our ProLab model enables property-level segmentation using descriptive prompts, enhancing interpretability and mirroring human-like understanding.