Relation Modeling and Distillation for Learning with Noisy Labels

TL;DR

This work tackles the challenge of learning with noisy labels, where label corruption degrades representation learning and model robustness. It introduces RMDNet, a plug-and-play framework with two modules: RM, which learns inter-sample relations through self-supervised learning (via a SimSiam-like setup) and constructs a relation graph, and RGRL, which distills this relational knowledge into the task network to calibrate representations of noisy samples. The approach combines a relation-based regularizer with standard cross-entropy loss, controlled by a hyperparameter K, and demonstrates improvements across CIFAR-10/100 under multiple noise types. Key contributions include the design of edge and node matching losses for graph distillation, comprehensive ablations, parameter analyses, and case studies illustrating improved separation of noisy and clean data. The findings suggest that incorporating relation modeling and knowledge distillation yields more robust representations and can be integrated with existing noisy-label methods to enhance performance in real-world settings.

Abstract

Learning with noisy labels has become an effective strategy for enhancing the robustness of models, which enables models to better tolerate inaccurate data. Existing methods either focus on optimizing the loss function to mitigate the interference from noise, or design procedures to detect potential noise and correct errors. However, their effectiveness is often compromised in representation learning due to the dilemma where models overfit to noisy labels. To address this issue, this paper proposes a relation modeling and distillation framework that models inter-sample relationships via self-supervised learning and employs knowledge distillation to enhance understanding of latent associations, which mitigate the impact of noisy labels. Specifically, the proposed method, termed RMDNet, includes two main modules, where the relation modeling (RM) module implements the contrastive learning technique to learn representations of all data, an unsupervised approach that effectively eliminates the interference of noisy tags on feature extraction. The relation-guided representation learning (RGRL) module utilizes inter-sample relation learned from the RM module to calibrate the representation distribution for noisy samples, which is capable of improving the generalization of the model in the inference phase. Notably, the proposed RMDNet is a plug-and-play framework that can integrate multiple methods to its advantage. Extensive experiments were conducted on two datasets, including performance comparison, ablation study, in-depth analysis and case study. The results show that RMDNet can learn discriminative representations for noisy data, which results in superior performance than the existing methods.

Figures (7)

• Figure 1: Motivation of the proposed RMDNet. Noisy labels diminish the effectiveness of representation learning during the training process. RMDNet firstly models the relation between the representations of samples without the need for label guidance. Secondly, RMDNet utilizes knowledge distillation to regularize representation learning to calibrate the distribution of representations for samples with noisy labels.
• Figure 2: The framework of the proposed RMDNet. It contains two main modules, where the Relation Modeling module employs self-supervised learning method to learn discriminative data representations. This can alleviate the interference of noise. The Relation-Guided Representation Learning module follows knowledge distillation to calibrate the representation distribution with noisy labels.
• Figure 3: The framework of Simsiam.
• Figure 4: Illustration of the Relation-Guided Representation Learning method. It contains two main constraint conditions, including the edge matching and the node matching.
• Figure 5: SimSiam is trained on the CIFAR-10 dataset before extracting and visualising the representations for the classes Ship and Truck.