From Open-Vocabulary to Vocabulary-Free Semantic Segmentation

TL;DR

Open-vocabulary segmentation struggles when class vocabularies must be specified in advance. This work proposes Vocabulary-Free Semantic Segmentation (VSS), a two-stage pipeline that uses an image tagging module to populate a broad semantic space $S$ and a CLIP-based segmentation decoder to produce pixel-level masks, with a cost volume $C$ defined by $C_{i,n} = \frac{D^V(i) \cdot D^L(n)}{|D^V(i)||D^L(n)|}$ and $K$ refinement steps via $T^{\text{sa}}$ and $T^{\text{ca}}$. Across benchmarks (ADE20K, Pascal Context, Pascal VOC), the approach outperforms prior OVSS methods and reveals the text encoder's critical role, while also highlighting the sensitivity to false negatives in tagging. The results demonstrate that vocabulary-free segmentation can realistically discover and describe unseen objects in real-world scenes, provided robust textual supervision and reliable taggers are in place.

Abstract

Open-vocabulary semantic segmentation enables models to identify novel object categories beyond their training data. While this flexibility represents a significant advancement, current approaches still rely on manually specified class names as input, creating an inherent bottleneck in real-world applications. This work proposes a Vocabulary-Free Semantic Segmentation pipeline, eliminating the need for predefined class vocabularies. Specifically, we address the chicken-and-egg problem where users need knowledge of all potential objects within a scene to identify them, yet the purpose of segmentation is often to discover these objects. The proposed approach leverages Vision-Language Models to automatically recognize objects and generate appropriate class names, aiming to solve the challenge of class specification and naming quality. Through extensive experiments on several public datasets, we highlight the crucial role of the text encoder in model performance, particularly when the image text classes are paired with generated descriptions. Despite the challenges introduced by the sensitivity of the segmentation text encoder to false negatives within the class tagging process, which adds complexity to the task, we demonstrate that our fully automated pipeline significantly enhances vocabulary-free segmentation accuracy across diverse real-world scenarios.

Figures (5)

• Figure 1: Traditional open-set understanding separates training and test classes, with an overlap allowing for unseen classes in testing. In Vision-Language Models (VLMs), trained on broad, internet-scale data, open-set recognition persists but is constrained by a finite query set. In a Vocabulary-Free setting, test classes are unrestricted, allowing any concept known to the VLM, introducing more realistic, complex variations beyond predefined vocabularies.
• Figure 2: Mask2Tag systems rely on class-agnostic masking models, where masks are associated with names. Oppositely, in our Tag2Mask pipeline, the image tagger supervises the text encoder, while the similarity between the text and vision embeddings produces the segmentation prediction. The segmentation module benefits from pretraining on a highly curated, close-set dataset.
• Figure 3: Comparison of segmentation results across ZeroSeg rewatbowornwong2023zero and Chicken-and-Egg (CaSED conti2024vocabulary and RAM zhang2024recognize), and ground-truth labels.
• Figure 4: Simulating missing classes or adding wrong ones over the OVSS baseline by assuming the labels are known at inference time.
• Figure 5: Ablation over different thresholds for the evaluation mapping.