The Evolution of Dataset Distillation: Toward Scalable and Generalizable Solutions

TL;DR

This survey analyzes advances in dataset distillation from 2023 to 2025, with a focus on scaling to ImageNet-scale data and beyond. It categorizes methods into gradient/trajectory/distribution matching, generative and decoupling approaches, and soft-label strategies, while addressing robustness, non-IID settings, and multimodal applications. Key contributions include the SRe2L decoupled optimization framework, lossless and diversity-enhancing distillation techniques, and diffusion-based generative methods that improve scalability and data quality. The paper also highlights persistent challenges in ultra-large-scale scalability, cross-architecture generalization, and standardized evaluation, and outlines concrete future directions for theory-driven and multimodal dataset distillation.

Abstract

Dataset distillation, which condenses large-scale datasets into compact synthetic representations, has emerged as a critical solution for training modern deep learning models efficiently. While prior surveys focus on developments before 2023, this work comprehensively reviews recent advances, emphasizing scalability to large-scale datasets such as ImageNet-1K and ImageNet-21K. We categorize progress into a few key methodologies: trajectory matching, gradient matching, distribution matching, scalable generative approaches, and decoupling optimization mechanisms. As a comprehensive examination of recent dataset distillation advances, this survey highlights breakthrough innovations: the SRe2L framework for efficient and effective condensation, soft label strategies that significantly enhance model accuracy, and lossless distillation techniques that maximize compression while maintaining performance. Beyond these methodological advancements, we address critical challenges, including robustness against adversarial and backdoor attacks, effective handling of non-IID data distributions. Additionally, we explore emerging applications in video and audio processing, multi-modal learning, medical imaging, and scientific computing, highlighting its domain versatility. By offering extensive performance comparisons and actionable research directions, this survey equips researchers and practitioners with practical insights to advance efficient and generalizable dataset distillation, paving the way for future innovations.

Figures (5)

• Figure 1: Illustration of dataset distillation wang2018dataset_arxiv2018. The original dataset is condensed into a synthetic dataset through a condensation process. Both the original and synthetic datasets are used to train randomly initialized models, and their performance is expected to be comparable.
• Figure 2: Overview of trajectory matching-based dataset distillation. The method aligns parameter trajectories between models trained on synthetic and real data.
• Figure 3: Overview of distribution matching in dataset distillation, where feature distributions are aligned to ensure the synthetic dataset effectively preserves the key characteristics of the original data.
• Figure 4: Overview of the three-stage SRe2L work yin2024squeeze_nips2024. The Squeeze Stage trains a randomly initialized model on the original dataset by minimizing cross-entropy loss. The Recover Stage refines the synthetic dataset by optimizing a combined cross-entropy and batch normalization (BN) loss with a well-converged model. Finally, the Relabel Stage evaluates performance by training another model on the synthetic dataset with soft labels, using cross-entropy loss.
• Figure 5: Overview of DD-RobustBench. Image from wu2024dd_arxiv2024.