CLIP Is Also a Good Teacher: A New Learning Framework for Inductive Zero-shot Semantic Segmentation

TL;DR

This work tackles Generalized Zero-shot Semantic Segmentation by transferring CLIP knowledge to a closed-set image encoder without adding test-time VLMs or extra modules. It introduces two modules, Global Learning Module (GLM) to align CLS-token–driven global semantics with dense features, and Pixel Learning Module (PLM) to generate semantically discriminative pseudo labels via multi-scale K-Means with mask fusion and to synthesize pseudo prototypes for unseen categories. The approach achieves state-of-the-art results on PASCAL VOC, COCO-Stuff, and PASCAL Context (notably in hIoU) with multiple backbones and is significantly faster in open-vocabulary settings. This framework demonstrates a practical, modular pipeline to realize open-vocabulary segmentation without costly VLM fine-tuning, enhancing zero-shot performance while leveraging unannotated regions through pseudo supervision.

Abstract

Generalized Zero-shot Semantic Segmentation aims to segment both seen and unseen categories only under the supervision of the seen ones. To tackle this, existing methods adopt the large-scale Vision Language Models (VLMs) which obtain outstanding zero-shot performance. However, as the VLMs are designed for classification tasks, directly adapting the VLMs may lead to sub-optimal performance. Consequently, we propose CLIP-ZSS (Zero-shot Semantic Segmentation), a simple but effective training framework that enables any image encoder designed for closed-set segmentation applied in zero-shot and open-vocabulary tasks in testing without combining with VLMs or inserting new modules. CLIP-ZSS consists of two key modules: Global Learning Module (GLM) and Pixel Learning Module (PLM). GLM is proposed to probe the knowledge from the CLIP visual encoder by pulling the CLS token and the dense features from the image encoder of the same image and pushing others apart. Moreover, to enhance the ability to discriminate unseen categories, PLM consisting of pseudo labels and weight generation is designed. To generate semantically discriminated pseudo labels, a multi-scale K-Means with mask fusion working on the dense tokens is proposed. In pseudo weight generation, a synthesizer generating pseudo semantic features for the unannotated area is introduced. Experiments on three benchmarks show large performance gains compared with SOTA methods.

Figures (8)

• Figure 1: Adopting by prompts or mask proposer
• Figure 2: Our methods
• Figure 4: The structure of CLIP-ZSS. Proposed methods are indicated by red dot lines.
• Figure 5: The overview of Pixel Learning Module (PLM).
• Figure 6: The structure of Synthesizer.