AlignZeg: Mitigating Objective Misalignment for Zero-shot Semantic Segmentation

TL;DR

AlignZeg markedly enhances zero-shot semantic segmentation, as shown by an average 3.8% increase in hIoU, primarily attributed to a 7.1% improvement in identifying unseen classes, and it is validated that the improvement comes from alleviating the objective misalignment issue.

Abstract

A serious issue that harms the performance of zero-shot visual recognition is named objective misalignment, i.e., the learning objective prioritizes improving the recognition accuracy of seen classes rather than unseen classes, while the latter is the true target to pursue. This issue becomes more significant in zero-shot image segmentation because the stronger (i.e., pixel-level) supervision brings a larger gap between seen and unseen classes. To mitigate it, we propose a novel architecture named AlignZeg, which embodies a comprehensive improvement of the segmentation pipeline, including proposal extraction, classification, and correction, to better fit the goal of zero-shot segmentation. (1) Mutually-Refined Proposal Extraction. AlignZeg harnesses a mutual interaction between mask queries and visual features, facilitating detailed class-agnostic mask proposal extraction. (2) Generalization-Enhanced Proposal Classification. AlignZeg introduces synthetic data and incorporates multiple background prototypes to allocate a more generalizable feature space. (3) Predictive Bias Correction. During the inference stage, AlignZeg uses a class indicator to find potential unseen class proposals followed by a prediction postprocess to correct the prediction bias. Experiments demonstrate that AlignZeg markedly enhances zero-shot semantic segmentation, as shown by an average 3.8% increase in hIoU, primarily attributed to a 7.1% improvement in identifying unseen classes, and we further validate that the improvement comes from alleviating the objective misalignment issue.

Figures (8)

• Figure 1: (a) Proposal features are aligned with class prototypes. "CLIP Text." represents CLIP Text Encoder. (b) Red boxes select the unseen classes. Yellow dashed boxes select the misclassified areas, e.g., "tree" (unseen) $\rightarrow$ "bush" (seen) ②, "road" (unseen) $\rightarrow$ "pavement" (unseen) ④. These errors show the objective misalignment issue.
• Figure 2: Overall framework. Our method mitigates the objective misalignment between semantic segmentation and zero-shot task through three main components: Mutually-Refined Proposal Extraction (MRPE), Generalization-Enhanced Proposal Classification (GEPC), and Proposal-based Bias Correction (PBC), the latter of which is applied during the inference process. Parameters in gray are fixed.
• Figure 3: Without GEPC, driven mainly by classification loss, the model focuses on seen classes, allowing them to dominate the feature space. GEPC introduces synthetic features and diverse backgrounds, helping provide a more generalizable feature space.
• Figure 4: Example of negative proposal selection. In an image with 3 seen classes, p5, p6, and p7 are selected as negative proposals since they are in the top-K (K=5) based on loss for all seen classes contained in the image.
• Figure 5: The $\mathrm{IoU}$ results for unseen classes on the COCO-Stuff. The dashed line represents the average result, i.e., $\mathrm{mIoU}(\mathcal{U})$.