Diffusion Models as Dataset Distillation Priors
Duo Su, Huyu Wu, Huanran Chen, Yiming Shi, Yuzhu Wang, Xi Ye, Jun Zhu
TL;DR
This work identifies representativeness as a missing piece in diffusion-model-based dataset distillation and answers how to leverage diffusion priors more effectively. It introduces Diffusion As Priors (DAP), a training-free framework that injects a kernel-based representativeness prior into the diffusion sampling process while preserving the intrinsic diversity and generalization priors of diffusion models. By defining a kernel-induced distance in feature space and guiding sampling with an energy-based representativeness term, DAP achieves state-of-the-art performance on ImageNet-1K and its subsets across DiT and Stable Diffusion backbones, with strong cross-architecture transfer and robustness to domain shifts. The approach offers a practical, modular, training-free pathway to higher-fidelity distilled datasets and builds a theoretical bridge between diffusion priors and dataset distillation objectives, with guidance for kernel choices and feature-layer selection.
Abstract
Dataset distillation aims to synthesize compact yet informative datasets from large ones. A significant challenge in this field is achieving a trifecta of diversity, generalization, and representativeness in a single distilled dataset. Although recent generative dataset distillation methods adopt powerful diffusion models as their foundation models, the inherent representativeness prior in diffusion models is overlooked. Consequently, these approaches often necessitate the integration of external constraints to enhance data quality. To address this, we propose Diffusion As Priors (DAP), which formalizes representativeness by quantifying the similarity between synthetic and real data in feature space using a Mercer kernel. We then introduce this prior as guidance to steer the reverse diffusion process, enhancing the representativeness of distilled samples without any retraining. Extensive experiments on large-scale datasets, such as ImageNet-1K and its subsets, demonstrate that DAP outperforms state-of-the-art methods in generating high-fidelity datasets while achieving superior cross-architecture generalization. Our work not only establishes a theoretical connection between diffusion priors and the objectives of dataset distillation but also provides a practical, training-free framework for improving the quality of the distilled dataset.