Dataset Distillers Are Good Label Denoisers In the Wild

TL;DR

This work rigorously evaluates three representative dataset distillation methods and reveals that dataset distillation effectively serves as a denoising tool in random noise scenarios but may struggle with structured asymmetric noise patterns, which can be absorbed into the distilled samples.

Abstract

Learning from noisy data has become essential for adapting deep learning models to real-world applications. Traditional methods often involve first evaluating the noise and then applying strategies such as discarding noisy samples, re-weighting, or re-labeling. However, these methods can fall into a vicious cycle when the initial noise evaluation is inaccurate, leading to suboptimal performance. To address this, we propose a novel approach that leverages dataset distillation for noise removal. This method avoids the feedback loop common in existing techniques and enhances training efficiency, while also providing strong privacy protection through offline processing. We rigorously evaluate three representative dataset distillation methods (DATM, DANCE, and RCIG) under various noise conditions, including symmetric noise, asymmetric noise, and real-world natural noise. Our empirical findings reveal that dataset distillation effectively serves as a denoising tool in random noise scenarios but may struggle with structured asymmetric noise patterns, which can be absorbed into the distilled samples. Additionally, clean but challenging samples, such as those from tail classes in imbalanced datasets, may undergo lossy compression during distillation. Despite these challenges, our results highlight that dataset distillation holds significant promise for robust model training, especially in high-privacy environments where noise is prevalent. The source code is available at https://github.com/Kciiiman/DD_LNL.

Figures (5)

• Figure 1: (a) for symmetric noise, existing dataset distillation methods serve as effective denoising tools. (b) structured patterns, such as asymmetric noise, can also be absorbed into the distilled samples, which hinders robust training.
• Figure 2: The validation performance over symmetric noise for CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets. The solid lines represent the accuracy trend as Image Per Class (IPC) increases, while the horizontal dashed lines of the same color indicate the performance of training on the full dataset at the corresponding noise rate.
• Figure 3: Visualization of images distilled from DATM on CIFAR-10 with one image per class.
• Figure 4: The validation performance over asymmetric noise for CIFAR-10, CIFAR-100. The solid lines represent the accuracy trend as Image Per Class (IPC) increases, while the horizontal dashed lines of the same color indicate the performance of training on the full dataset at the corresponding noise rate
• Figure 5: The results for CIFAR-10N/CIFAR-100N. The solid lines illustrate the trend of accuracy as Image Per-Class (IPC) increases. The horizontal dashed lines of the same color indicate the evaluation of training on the full dataset under the corresponding noise rate.