Scaling up Multi-domain Semantic Segmentation with Sentence Embeddings

TL;DR

This work tackles open-set and cross-domain semantic segmentation by replacing class labels with sentence embeddings generated from descriptive text via CLIP, enabling zero-shot segmentation and the merging of diverse datasets. It builds a large heterogeneous training corpus of about 2 million images from 9 datasets by encoding 238 labels with sentence descriptions from Wikipedia, and trains with a three-loss framework to handle noisy and weak labels. The method achieves state-of-the-art zero-shot performance on several benchmarks and, after fine-tuning, surpasses previous supervised methods on NYU v2 and Pascal Context, while also enabling segmentation of unseen labels. Beyond segmentation, the approach boosts downstream tasks such as monocular depth estimation and instance segmentation via pseudo-labels and CLIP-based distillation, highlighting the practical impact of language-guided, multi-domain learning.

Abstract

We propose an approach to semantic segmentation that achieves state-of-the-art supervised performance when applied in a zero-shot setting. It thus achieves results equivalent to those of the supervised methods, on each of the major semantic segmentation datasets, without training on those datasets. This is achieved by replacing each class label with a vector-valued embedding of a short paragraph that describes the class. The generality and simplicity of this approach enables merging multiple datasets from different domains, each with varying class labels and semantics. The resulting merged semantic segmentation dataset of over 2 Million images enables training a model that achieves performance equal to that of state-of-the-art supervised methods on 7 benchmark datasets, despite not using any images therefrom. By fine-tuning the model on standard semantic segmentation datasets, we also achieve a significant improvement over the state-of-the-art supervised segmentation on NYUD-V2 and PASCAL-context at 60% and 65% mIoU, respectively. Based on the closeness of language embeddings, our method can even segment unseen labels. Extensive experiments demonstrate strong generalization to unseen image domains and unseen labels, and that the method enables impressive performance improvements in downstream applications, including depth estimation and instance segmentation.

Figures (9)

• Figure 1: Examples of segmenting zero-shot labels on web images. The labels 'duck' (first row) and 'deer' (second row) are not in the training data for Mseg MSeg_2020_CVPR, or the proposed method. Note that our proposed method is able to segment unseen labels successfully, however.
• Figure 2: Our framework. We merge multiple datasets together and encode all labels to embeddings using a language model. The semantic segmentation model is enforced to learn pixel-wise embeddings. During testing, the cosine similarity between the predicted embedding and the search embedding space is calculated. The output class is the most similar one in the label space.
• Figure 3: A visualisation of the pairwise distances between label embeddings for (a) single word class labels and (b) sentences. The sentence embeddings better reflect the underlying semantic similarity between classes. For example, 'Animals' classes are close to each other but far from those in 'People' and 'Transport'. Capturing and exploiting these semantic similarities in diverse datasets can improve both zero-shot, and fully supervised segmentation performance.
• Figure 4: Distillation pipeline for weakly-labeled data. We use the clip classification model to obtain the vision embedding for the bounding box regions. It is applied to supervise the predicted bounding box embedding.
• Figure 5: Examples of noisy annotations from OpenImages (see red arrows regions). Left to right: masks for lion, bowl, billiard table, and airplane.