Are Large-scale Soft Labels Necessary for Large-scale Dataset Distillation?

TL;DR

This paper discovers that the high within-class similarity in condensed datasets necessitates the use of large-scale soft labels, and introduces class-wise supervision during the image synthesizing process by batching the samples within classes, instead of across classes, to increase within-class diversity and reduce the size of required soft labels.

Abstract

In ImageNet-condensation, the storage for auxiliary soft labels exceeds that of the condensed dataset by over 30 times. However, are large-scale soft labels necessary for large-scale dataset distillation? In this paper, we first discover that the high within-class similarity in condensed datasets necessitates the use of large-scale soft labels. This high within-class similarity can be attributed to the fact that previous methods use samples from different classes to construct a single batch for batch normalization (BN) matching. To reduce the within-class similarity, we introduce class-wise supervision during the image synthesizing process by batching the samples within classes, instead of across classes. As a result, we can increase within-class diversity and reduce the size of required soft labels. A key benefit of improved image diversity is that soft label compression can be achieved through simple random pruning, eliminating the need for complex rule-based strategies. Experiments validate our discoveries. For example, when condensing ImageNet-1K to 200 images per class, our approach compresses the required soft labels from 113 GB to 2.8 GB (40x compression) with a 2.6% performance gain. Code is available at: https://github.com/he-y/soft-label-pruning-for-dataset-distillation

Figures (8)

• Figure 1: The relationship between performance and total storage of auxiliary information needed. Our method achieves SOTA performance with fewer soft labels than images.
• Figure 2: Visual comparison between SRe$^2$L and the proposed method. The classes are hammer shark (top), pineapple (middle), and pomegranate (bottom). Our method is more visually diverse.
• Figure 3: MMD visualization.
• Figure 4: Illustration of existing methods (left, grey) and the proposed method (right, blue). Existing methods (i.e., SRe$^2$L, CDA) independently generate along the IPC (Image-Per-Class) dimension, causing a high similarity between images of the same class. The proposed method allows images of the same class to collaborate, leaving different classes naturally independent. In addition, synthetic images are updated under class-wise supervision. The classification loss is omitted for simplicity.
• Figure 5: Illustration of two random processes in label pruning with improved label pool. First, we need a smaller soft label pool due to the storage budget. We can conduct pruning at two levels: (1) epoch-level and (2) batch-level. Batch-level pruning can provide a more diverse label pool since augmentations (e.g., Mixup or CutMix) are different across batches. The illustrated pruning ratio is 25%; the crossed-out labels denote the pruned labels, and the remaining form the label pool. Second, we randomly sample soft labels for model training.

Theorems & Definitions (2)

• Proposition 1
• Proposition 2