Taming Diffusion for Dataset Distillation with High Representativeness

TL;DR

There is a rising need for compact yet effective training data. The authors introduce D$^3$HR, a diffusion-based dataset distillation framework that maps VAE latent space to a high-normality Gaussian domain via deterministic DDIM inversion, enabling accurate distribution matching; it then uses a group-sampling strategy to select a highly representative latent subset before decoding to images. The approach achieves state-of-the-art accuracy across multiple datasets and model architectures, with strong cross-architecture generalization and notable storage efficiency by storing only Gaussian parameters and diffusion weights. Key contributions include the domain-mapping mechanism, a principled distribution-matching objective, and a scalable, parallelizable subset search, all demonstrated through extensive ablations and analyses. Overall, D$^3$HR provides a practical, architecture-agnostic path to high-quality distilled datasets for diverse learners.

Abstract

Recent deep learning models demand larger datasets, driving the need for dataset distillation to create compact, cost-efficient datasets while maintaining performance. Due to the powerful image generation capability of diffusion, it has been introduced to this field for generating distilled images. In this paper, we systematically investigate issues present in current diffusion-based dataset distillation methods, including inaccurate distribution matching, distribution deviation with random noise, and separate sampling. Building on this, we propose D^3HR, a novel diffusion-based framework to generate distilled datasets with high representativeness. Specifically, we adopt DDIM inversion to map the latents of the full dataset from a low-normality latent domain to a high-normality Gaussian domain, preserving information and ensuring structural consistency to generate representative latents for the distilled dataset. Furthermore, we propose an efficient sampling scheme to better align the representative latents with the high-normality Gaussian distribution. Our comprehensive experiments demonstrate that D^3HR can achieve higher accuracy across different model architectures compared with state-of-the-art baselines in dataset distillation. Source code: https://github.com/lin-zhao-resoLve/D3HR.

Figures (12)

• Figure 1: t-SNE visualization of the low-normality VAE space and high-normality noise space for class “Goldfish". The blue contour lines are the probability density curves of the distribution using kernel density estimation, highlighting the structure and concentration of the latents (blue dots). ★ in (b) marks the $10$ representative latents generated in the noise space, corresponding to ★ in (a) after DDIM sampling, which preserves the structure of VAE space and concentrates in high-density regions.
• Figure 2: Illustration of proposed D$^3$HR framework. With the latents from the VAE, DDIM inversion is applied to map the latent embeddings to a Gaussian domain with better normality, which can further be matched to a Gaussian distribution. Then, we follow \ref{['subsec:gaussian sampling']} to sample representative latents based on different IPC requirements, and generate true images through DDIM sampling.
• Figure 3: t-SNE visualization of the representative latents in VAE space generated by \ref{['equ:ddpm']} and D$^3$HR for class “Goldfish". It can be observed that the latents accurately represent the VAE distribution in D$^3$HR.
• Figure 4: Validation of the effectiveness and stability of group sampling on ImageNet-1K. For our D$^3$HR and Base-RS, we generate 3 distilled datasets at each IPC. Each dataset undergoes 3 rounds of validation, resulting in 9 data points per box plot.
• Figure 5: The accuracy variation under different inversion timesteps for ImageNet-1K, $\text{IPC}=10$.

Theorems & Definitions (4)

• Lemma 3.1
• Lemma 4.1
• Lemma 1.1
• Lemma 1.2