Generalized Class Discovery in Instance Segmentation

TL;DR

This work proposes an instance-wise temperature assignment (ITA) method for contrastive learning and class-wise reliability criteria for pseudo-labels and introduces an efficient soft attention module to encode object-specific representations for GCD.

Abstract

This work addresses the task of generalized class discovery (GCD) in instance segmentation. The goal is to discover novel classes and obtain a model capable of segmenting instances of both known and novel categories, given labeled and unlabeled data. Since the real world contains numerous objects with long-tailed distributions, the instance distribution for each class is inherently imbalanced. To address the imbalanced distributions, we propose an instance-wise temperature assignment (ITA) method for contrastive learning and class-wise reliability criteria for pseudo-labels. The ITA method relaxes instance discrimination for samples belonging to head classes to enhance GCD. The reliability criteria are to avoid excluding most pseudo-labels for tail classes when training an instance segmentation network using pseudo-labels from GCD. Additionally, we propose dynamically adjusting the criteria to leverage diverse samples in the early stages while relying only on reliable pseudo-labels in the later stages. We also introduce an efficient soft attention module to encode object-specific representations for GCD. Finally, we evaluate our proposed method by conducting experiments on two settings: COCO$_{half}$ + LVIS and LVIS + Visual Genome. The experimental results demonstrate that the proposed method outperforms previous state-of-the-art methods.

Figures (4)

• Figure 1: Overview of the proposed framework during training. We first train a class-agnostic instance segmentation network $f_o(\cdot)$ and apply it to unlabeled images to generate class-agnostic instance masks. Then, we train the GCD model $f_d(\cdot)$ using unlabeled object images and labeled object images to discover novel classes in the unlabeled data. Finally, we train an instance segmentation network $f_s(\cdot)$ using the labeled data and the unlabeled images with pseudo-labels.
• Figure 2: $t$-SNE visualization of two semantically close classes. Green: single head class ('book'); Blue: single tail class ('booklet').
• Figure 3: Visualization of the pairwise affinity between the white cross marked location and other pixels.
• Figure 4: Qualitative results. (a) and (b) are COCO$_{half}$ + LVIS setting; (c) and (d) are LVIS + VG setting.