Incremental Label Distribution Learning with Scalable Graph Convolutional Networks

TL;DR

A New-label-aware Gradient Compensation Loss to speed up the learning of new labels and represent inter-label relationships as a graph to reduce the time required to reconstruct inter-label relationships is developed.

Abstract

Label Distribution Learning (LDL) is an effective approach for handling label ambiguity, as it can analyze all labels at once and indicate the extent to which each label describes a given sample. Most existing LDL methods consider the number of labels to be static. However, in various LDL-specific contexts (e.g., disease diagnosis), the label count grows over time (such as the discovery of new diseases), a factor that existing methods overlook. Learning samples with new labels directly means learning all labels at once, thus wasting more time on the old labels and even risking overfitting the old labels. At the same time, learning new labels by the LDL model means reconstructing the inter-label relationships. How to make use of constructed relationships is also a crucial challenge. To tackle these challenges, we introduce Incremental Label Distribution Learning (ILDL), analyze its key issues regarding training samples and inter-label relationships, and propose Scalable Graph Label Distribution Learning (SGLDL) as a practical framework for implementing ILDL. Specifically, in SGLDL, we develop a New-label-aware Gradient Compensation Loss to speed up the learning of new labels and represent inter-label relationships as a graph to reduce the time required to reconstruct inter-label relationships. Experimental results on the classical LDL dataset show the clear advantages of unique algorithms and illustrate the importance of a dedicated design for the ILDL problem.

Figures (5)

• Figure 1: A toy example to show how the Label Distribution Learning model changes the prediction value of the old labels after learning a new label. As we can see, SLL / MLL and LDL judge the same label distribution very differently, predisposing their corresponding models to react very differently when learning new labels.
• Figure 2: An example of predicting age distribution. The model outputs different distributions for the same image before and after incremental learning in the model's label space.
• Figure 3: The variation of parameter modification caused by different classifiers in solving the ILDL problem. The deep red region indicates the parameters that need to be modified. Traditional fully connected layer need to modify all parameters due to the output characteristics of LDL. But for SGLDL, only the parameter values corresponding to the new label need to be modified, as only the knowledge in this part changed.
• Figure 4: The framework of SGLDL involves feeding samples into a feature extractor (CNN) to extract features while generating the corresponding GCN through SCM to ensure effective incremental learning. The SCM is saved, and each incremental learning step extends the classifier (GCN) based on the previous SCM.
• Figure 5: SCM visualization on IMDB-WIKI dataset