Training-Free Semantic Segmentation via LLM-Supervision

TL;DR

This work tackles open-vocabulary semantic segmentation with no extra training by leveraging a large language model to generate fine-grained subclasses for each class. It integrates LLM-generated subclass descriptors into a training-free CLIP-based segmentation framework (SimSeg) using locality-driven alignment, upsampling, and CRF refinements, followed by ensembling across subclass descriptors to improve coverage of image content. Key contributions include: (i) automated generation of discriminative subclasses via GPT-3 with few-shot prompts, (ii) a training-free segmentation pipeline that leverages textual-subclass supervision, and (iii) an ensembling strategy that fuses multiple subclass maps for better accuracy. Experiments on PASCAL VOC 2012, PASCAL Context, and COCO-Stuff demonstrate consistent gains over traditional text-supervised baselines, with ablations showing the impact of subclass quality, number of generated subclasses, and template prompts. The approach reduces labeling needs while enhancing segmentation fidelity, offering a practical path to open-vocabulary semantic segmentation.

Abstract

Recent advancements in open vocabulary models, like CLIP, have notably advanced zero-shot classification and segmentation by utilizing natural language for class-specific embeddings. However, most research has focused on improving model accuracy through prompt engineering, prompt learning, or fine-tuning with limited labeled data, thereby overlooking the importance of refining the class descriptors. This paper introduces a new approach to text-supervised semantic segmentation using supervision by a large language model (LLM) that does not require extra training. Our method starts from an LLM, like GPT-3, to generate a detailed set of subclasses for more accurate class representation. We then employ an advanced text-supervised semantic segmentation model to apply the generated subclasses as target labels, resulting in diverse segmentation results tailored to each subclass's unique characteristics. Additionally, we propose an assembly that merges the segmentation maps from the various subclass descriptors to ensure a more comprehensive representation of the different aspects in the test images. Through comprehensive experiments on three standard benchmarks, our method outperforms traditional text-supervised semantic segmentation methods by a marked margin.

Figures (10)

• Figure 1: Overview of the proposed approach. Initially, we start from an LLM to generate subclasses for a given superclass in the test phase. Subsequently, we utilize a pre-existing semantic segmentation model SimSeg, initially supervised by superclasses, and apply the LLM-generated subclasses for supervision. To obtain the final results, we ensemble the results from various subclasses(blue areas). Compared with the traditional superclass-supervised method (purple area), our LLM-supervision (yellow areas) achieves distinguishable and precise segmentation results.
• Figure 2: Training-free semantic segmentation via LLM supervision. Initially, the LLM generates four subcategories within the 'person' category. These generated prompts and a template are then processed through a text encoder to derive textual features, while image features are extracted using an image encoder. The next step involves calculating the similarity $\mathbf{A}$ between the textual features, which are organized and aligned with a locality-driven approach, and the image features. This similarity is utilized to create rough masks for each subclass. For greater accuracy, these masks are refined using upsampling and a conditional random field (CRF). Finally, an ensemble technique is employed to merge the rough masks of the four subclasses, resulting in the final prediction.
• Figure 2: Segmentation Results with Our LLM-Supervision. The use of subclass textual representations leads to more informative and precise segmentation outcomes compared to those achieved with superclass textual representations.
• Figure 3: Ensembling of subclasses descriptors. We integrate descriptors from different subclasses to achieve more accurate segmentation results by utilizing the attention weights between chosen textual and visual features.
• Figure 4: Segmentation Results with Our LLM-Supervision. The use of subclass textual representations leads to more informative and precise segmentation outcomes compared to those achieved with superclass textual representations.