Solving the Catastrophic Forgetting Problem in Generalized Category Discovery

TL;DR

This work designs a novel learning approach, LegoGCD, which is seamlessly integrated into previous methods to enhance the discrimination of novel classes while maintaining performance on previously encountered known classes and effectively addresses the known category forgetting issue across all datasets.

Abstract

Generalized Category Discovery (GCD) aims to identify a mix of known and novel categories within unlabeled data sets, providing a more realistic setting for image recognition. Essentially, GCD needs to remember existing patterns thoroughly to recognize novel categories. Recent state-of-the-art method SimGCD transfers the knowledge from known-class data to the learning of novel classes through debiased learning. However, some patterns are catastrophically forgot during adaptation and thus lead to poor performance in novel categories classification. To address this issue, we propose a novel learning approach, LegoGCD, which is seamlessly integrated into previous methods to enhance the discrimination of novel classes while maintaining performance on previously encountered known classes. Specifically, we design two types of techniques termed as Local Entropy Regularization (LER) and Dual-views Kullback Leibler divergence constraint (DKL). The LER optimizes the distribution of potential known class samples in unlabeled data, thus ensuring the preservation of knowledge related to known categories while learning novel classes. Meanwhile, DKL introduces Kullback Leibler divergence to encourage the model to produce a similar prediction distribution of two view samples from the same image. In this way, it successfully avoids mismatched prediction and generates more reliable potential known class samples simultaneously. Extensive experiments validate that the proposed LegoGCD effectively addresses the known category forgetting issue across all datasets, eg, delivering a 7.74% and 2.51% accuracy boost on known and novel classes in CUB, respectively. Our code is available at: https://github.com/Cliffia123/LegoGCD.

Figures (7)

• Figure 1: Visualization of the accuracy results in unlabeled dataset on CUB dataset wah2011caltech during training. (a) shows a decrease in the accuracy of known (Old) classes (green) in the baseline as the accuracy of novel (New) classes (orange) increases. (b) demonstrates that LegoGCD solves the catastrophic forgetting problem and surpasses the baseline by a significant margin of $7.74$.
• Figure 2: Illustration of our proposed LegoGCD. LegoGCD is mainly composed of SimGCD and our proposed LER and DKL. (a) Representation learning and Mean Entropy in SimGCD (\ref{['preliminaries']}). (b) Local Entropy Regularization (LER) (\ref{['LER']}) for discovering potential known samples in unlabeled data and preserving the knowledge of known classes. (c) Dual-views Kullback-Leibler divergence (DKL) (\ref{['DKL']}) to ensure consistent predictions for two view samples.
• Figure 3: Comparison of potential known samples in SimGCD wen2023parametric and LegoGCD. (a) LegoGCD recognizes almost 10 more high-confidence samples than SimGCD in the end on the FGVC-Aircraft dataset. (b) LegoGCD with LER produces more high-confidence known samples in various generic and fine-grained datasets compared to SimGCD without LER.
• Figure 4: Step by step, we integrate LER and DKL into the baseline SimGCD wen2023parametric. Initially, the addition of LER increases accuracy in the "Old" category while decreasing accuracy in the "New" category. Subsequently, the introduction of a Margin-aware Pattern (MAP) widens margins between novel categories, ultimately achieving the best performance when embedding with DKL.
• Figure 5: Data details of traditional classification and Generalized Category Discovery.