Bridge the Gap Between Visual and Linguistic Comprehension for Generalized Zero-shot Semantic Segmentation

TL;DR

This work tackles generalized zero-shot semantic segmentation by addressing the limitations of single semantic representations. It introduces DeVLMatch, a decoupled vision-language framework with SPMatch and CSMatch that align fine-grained object parts and states across visual and linguistic graphs via graph matching and Sinkhorn-based sparsification. The approach yields state-of-the-art results on PASCAL VOC, COCO-Stuff, and CATARACTS, and demonstrates robustness in medical segmentation and extensions to new backbones and SAM. Ablation studies validate the benefit of decoupling parts/states and the effectiveness of the proposed graph-pairings for knowledge transfer to unseen classes. The method offers a flexible, interpretable embedding-based alternative that scales to extended class sets while maintaining training-time efficiency gains through plug-in modules.

Abstract

Generalized zero-shot semantic segmentation (GZS3) aims to achieve the human-level capability of segmenting not only seen classes but also novel class regions unseen in the training data through introducing the bridge of semantic representations, e.g., word vector. While effective, the way of utilizing one semantic representation to associate the corresponding class and to enable the knowledge transfer from seen to unseen classes is insufficient as well as incompatible with human cognition. Inspired by the observation that humans often use some `part' and `state' information to comprehend the seen objects and imagine unseen classes, we decouple each class into detailed descriptions, including object parts and states. Based on the decoupling formulation, we propose a Decoupled Vision-Language Matching (DeVLMatch) framework, composed of spatial-part (SPMatch) and channel-state (CSMatch) matching modules, for GZS3. In SPMatch, we comprehend objects with spatial part information from both visual and linguistic perspectives and perform graph matching to bridge the gap. In CSMatch, states of objects from the linguistic perspective are matched to compatible channel information from the visual perspective. By decoupling and matching objects across visual and linguistic comprehension, we can explicitly introspect the relationship between seen and unseen classes in fine-grained object part and state levels, thereby facilitating the knowledge transfer from seen to unseen classes in visual space. The proposed DeVLMatch framework surpasses the previous GZS3 methods on standard benchmarks, including PASCAL VOC, COCO-Stuff, and CATARACTS, demonstrating its effectiveness.

Figures (12)

• Figure 1: Existing GZS3 methods usually use a single word vector or an attribute as semantic representation to associate each class, such as the 'cat' class (a). In contrast, inspired by the linguistic comprehension of human, we decouple the 'cat' class into detailed descriptions, including parts (b1) and states (b2) of the object, such as 'cateye' and 'browncat', which enables us to introspect the shared knowledge among classes and facilitate the generalizability of model on unseen classes, such as 'bird'.
• Figure 2: Illustration of our Decoupled Vision-Language Matching (DeVLMatch) framework with (a) SPMatch and (b) CSMatch modules. The gray part is baseline ZegFormer ding2022decoupling.
• Figure 3: Illustration of our spatial-part matching (SPMatch) module.
• Figure 4: Illustration of our channel-state matching (CSMatch) module.
• Figure 5: Sensitivity analysis on hyper parameters, including spatial query number $k$, channel split number $R$, weighting coefficients $\alpha$, $\beta$.