Rectifying Soft-Label Entangled Bias in Long-Tailed Dataset Distillation

TL;DR

This work tackles soft-label bias in long-tailed dataset distillation by deriving an imbalance-aware generalization bound and identifying two entangled soft-label biases arising from the distillation model and distilled images. It introduces ADSA, a lightweight, post-hoc Adaptive Soft-label Alignment module that calibrates soft labels without altering training pipelines, leading to robust gains across multiple distillation methods and datasets. Empirically, ADSA yields tail-class improvements up to 11.8 percentage points and raises overall accuracy (e.g., 41.4% on ImageNet-LT with IPC=50), demonstrating strong generalization under limited label budgets. The approach offers a data-centric, plug-and-play solution for long-tailed recognition, with theoretical backing and practical efficacy across scales and settings.

Abstract

Dataset distillation compresses large-scale datasets into compact, highly informative synthetic data, significantly reducing storage and training costs. However, existing research primarily focuses on balanced datasets and struggles to perform under real-world long-tailed distributions. In this work, we emphasize the critical role of soft labels in long-tailed dataset distillation and uncover the underlying mechanisms contributing to performance degradation. Specifically, we derive an imbalance-aware generalization bound for model trained on distilled dataset. We then identify two primary sources of soft-label bias, which originate from the distillation model and the distilled images, through systematic perturbation of the data imbalance levels. To address this, we propose ADSA, an Adaptive Soft-label Alignment module that calibrates the entangled biases. This lightweight module integrates seamlessly into existing distillation pipelines and consistently improves performance. On ImageNet-1k-LT with EDC and IPC=50, ADSA improves tail-class accuracy by up to 11.8% and raises overall accuracy to 41.4%. Extensive experiments demonstrate that ADSA provides a robust and generalizable solution under limited label budgets and across a range of distillation techniques. Code is available at: https://github.com/j-cyoung/ADSA_DD.git.

Figures (4)

• Figure 1: Overview of the experimental framework and modules. (a) The conventional dataset distillation pipeline utilizes a single model for both image and label generation. (b) The perturbation analysis framework employs two separate models: one for image synthesis and another for soft label generation. Three configurations indicate different combination of balanced/imbalanced dataset. We then perturb the imbalance levels to observe the resulting performance. (c) The proposed adaptive soft-label alignment module(ADSA). Symbols $\bar{y}$, $\Delta y$, and $\tilde{y}$ denote the average confidence, confidence adjustment by different levels, and the calibrated average confidence, respectively. We optimize $\Delta y$ to get the most uniform $\tilde{y}$ across classes, and use it to calibrate the soft labels. The operator $\oplus$ indicates addition in logit space; soft labels are shown here for visualization clarity.
• Figure 2: Effect of the number of images across head and tail classes on confidence, accuracy, and entropy. (a): Accuracy trend on tail classes. (b): Entropy of soft labels for tail classes. (c) and (d): Confidence scores for tail and head classes with increasing tail samples.
• Figure 3: Left: Class-wise accuracy under different imbalance factors (IF) and images per class (IPC). Our method consistently improves overall and tail-class performance. Right: (a) Accuracy under varying IPC; (b) Accuracy under varying IF; (c) Class-wise confidence difference between original and distilled images; (d) Per-class soft-label confidence distributions. In panel (c) and (d)"imb"/"bal" denote soft labels from models trained on imbalanced/balanced original data; "cal" indicates calibrated soft labels 'imb' from the imbalanced model. Dashed lines represent raw outputs; solid lines are EMA-smoothed results.
• Figure 4: Left: Distribution of selected $\tau$ across different datasets. The value of $\tau$ varies within the same dataset due to differences in IPC, IF, and the random seed used in training. Right: t-SNE visualization of long-tailed and distilled datasets under different imbalance factors (IF). The top row shows the original long-tailed dataset distributions, which remain well-separated across different IF values. The bottom row presents the corresponding distilled datasets, where class clusters become increasingly compressed and less distinguishable as the imbalance factor increases.

Theorems & Definitions (1)

• Theorem 3.1