Addressing Imbalance for Class Incremental Learning in Medical Image Classification

TL;DR

The paper tackles catastrophic forgetting in class incremental learning for medical image classification caused by data imbalance. It introduces two plug-in losses: the CIL-balanced classification loss $L_{cbc}$, which uses logit adjustment to mitigate bias toward old and minority classes, and the distribution margin loss $L_{dm}$, which pushes distributions of old and new classes apart while promoting intra-class compactness; both are combined with knowledge distillation in the overall objective $L_{all}$. The method defines category frequencies $r_c$, logit adjustments with $v_{y_i,j}$ and a scale factor $\\gamma_c$, and models class distributions with $\\hat{w}_c = w_c + \eta \\hat{r}_c$, where $\\hat{r}_c = q_c / \\sum q_i$, to formulate $L_{dm}$. Extensive experiments on CCH5000, HAM10000, and EyePACS show state-of-the-art performance in both average accuracy and forgetting across multiple incremental settings, highlighting improved resilience to imbalance in medical CIL with practical applicability.

Abstract

Deep convolutional neural networks have made significant breakthroughs in medical image classification, under the assumption that training samples from all classes are simultaneously available. However, in real-world medical scenarios, there's a common need to continuously learn about new diseases, leading to the emerging field of class incremental learning (CIL) in the medical domain. Typically, CIL suffers from catastrophic forgetting when trained on new classes. This phenomenon is mainly caused by the imbalance between old and new classes, and it becomes even more challenging with imbalanced medical datasets. In this work, we introduce two simple yet effective plug-in methods to mitigate the adverse effects of the imbalance. First, we propose a CIL-balanced classification loss to mitigate the classifier bias toward majority classes via logit adjustment. Second, we propose a distribution margin loss that not only alleviates the inter-class overlap in embedding space but also enforces the intra-class compactness. We evaluate the effectiveness of our method with extensive experiments on three benchmark datasets (CCH5000, HAM10000, and EyePACS). The results demonstrate that our approach outperforms state-of-the-art methods.

Figures (7)

• Figure 1: Overview of the class incremental learning setting in medical image classification. During the incremental process, the training data is only provided for the current classes, while the data from previous steps is not accessible. At each step, the model is required to perform classification for all the classes seen so far.
• Figure 2: (a) The vanilla margin loss forces the cosine similarity between $h_a$ and $w_p$ to be larger than that between $h_a$ and $w_n$ without considering the distribution separation. (b) Our distribution margin loss aims to push $h_a$ away from the distribution of the negative class instead of just $w_n$, thus mitigating feature space overlap. (c) The vanilla margin loss fails to minimize the intra-class distance adequately, which may result in $h_a$ being distant from the center of its ground-truth class. (d) The distribution margin loss ensures that $h_a$ remains within its corresponding class distribution, enhancing intra-class compactness.
• Figure 3: Accuracy at each step on CCH5000, HAM10000, and EyePACS.
• Figure 4: Forgetting at each step on CCH5000, HAM10000, and EyePACS.
• Figure 5: The t-SNE visualization of feature distributions of w/o margin loss (left), margin ranking loss (middle), and our distribution margin loss (right) on the CCH5000 dataset.