Difficulty-guided Sampling: Bridging the Target Gap between Dataset Distillation and Downstream Tasks

TL;DR

This work tackles the target gap between dataset distillation objectives and downstream tasks by introducing difficulty-guided sampling (DGS), a post-stage sampling module that leverages the difficulty distribution of the original data to select distilled samples from an image pool. It also proposes difficulty-aware guidance (DAG) to directly generate samples that follow the desired difficulty distribution. Through extensive experiments on multiple ImageNet subsets and models, DGS consistently improves classification performance and, when combined with strong baselines like Minimax, achieves state-of-the-art results, while DAG demonstrates the broader potential of difficulty-based generation. The findings highlight the value of task-specific information in distillation, suggesting avenues for extending the approach to other domains and tasks beyond image classification, such as object detection or privacy-preserving settings.

Abstract

In this paper, we propose difficulty-guided sampling (DGS) to bridge the target gap between the distillation objective and the downstream task, therefore improving the performance of dataset distillation. Deep neural networks achieve remarkable performance but have time and storage-consuming training processes. Dataset distillation is proposed to generate compact, high-quality distilled datasets, enabling effective model training while maintaining downstream performance. Existing approaches typically focus on features extracted from the original dataset, overlooking task-specific information, which leads to a target gap between the distillation objective and the downstream task. We propose leveraging characteristics that benefit the downstream training into data distillation to bridge this gap. Focusing on the downstream task of image classification, we introduce the concept of difficulty and propose DGS as a plug-in post-stage sampling module. Following the specific target difficulty distribution, the final distilled dataset is sampled from image pools generated by existing methods. We also propose difficulty-aware guidance (DAG) to explore the effect of difficulty in the generation process. Extensive experiments across multiple settings demonstrate the effectiveness of the proposed methods. It also highlights the broader potential of difficulty for diverse downstream tasks.

Figures (8)

• Figure 1: Illustration of the workflow of non-generative and generative dataset distillation methods, summarized from current methods. The blue areas show the part that is repeated when generating multiple images.
• Figure 2: Kernel density estimates of the difficulty distributions for the original dataset and the image pool generated by Minimax gu2024minimax. The left figure shows the results across the entire dataset. The second figure shows class-wise density estimates, and the horizontal axis range is adjusted to around $0 \sim 0.4$ for better readability.
• Figure 3: Workflow of DGS. The overall processing is highlighted in green. The generation of the image pool is shown within the blue background. The sampling process is indicated within the yellow background. Distribution smoothing is applied to the difficulty distributions of both the original dataset and the image pool. The sampling distribution scales from the difficulty distribution of the original dataset.
• Figure 4: Workflow of DAG. The images in the original dataset are encoded into the latent space. Hierarchical clustering is conducted following the sampling distribution. Clustering centers are used to guide the denoising process.
• Figure 5: Images from the original dataset and the DGS-sampled dataset with difficulty scores. The sampled dataset follows the difficulty distribution of the original dataset. Images with similar difficulty scores exhibit shared characteristics.