Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Data-Efficient Generation for Dataset Distillation

Zhe Li, Weitong Zhang, Sarah Cechnicka, Bernhard Kainz

TL;DR

This work trains a class conditional latent diffusion model capable of generating realistic synthetic images with labels and demonstrates that models can be effectively trained using only a small set of synthetic images and evaluated on a large real test set.

Abstract

While deep learning techniques have proven successful in image-related tasks, the exponentially increased data storage and computation costs become a significant challenge. Dataset distillation addresses these challenges by synthesizing only a few images for each class that encapsulate all essential information. Most current methods focus on matching. The problems lie in the synthetic images not being human-readable and the dataset performance being insufficient for downstream learning tasks. Moreover, the distillation time can quickly get out of bounds when the number of synthetic images per class increases even slightly. To address this, we train a class conditional latent diffusion model capable of generating realistic synthetic images with labels. The sampling time can be reduced to several tens of images per seconds. We demonstrate that models can be effectively trained using only a small set of synthetic images and evaluated on a large real test set. Our approach achieved rank \(1\) in The First Dataset Distillation Challenge at ECCV 2024 on the CIFAR100 and TinyImageNet datasets.

Data-Efficient Generation for Dataset Distillation

TL;DR

This work trains a class conditional latent diffusion model capable of generating realistic synthetic images with labels and demonstrates that models can be effectively trained using only a small set of synthetic images and evaluated on a large real test set.

Abstract

While deep learning techniques have proven successful in image-related tasks, the exponentially increased data storage and computation costs become a significant challenge. Dataset distillation addresses these challenges by synthesizing only a few images for each class that encapsulate all essential information. Most current methods focus on matching. The problems lie in the synthetic images not being human-readable and the dataset performance being insufficient for downstream learning tasks. Moreover, the distillation time can quickly get out of bounds when the number of synthetic images per class increases even slightly. To address this, we train a class conditional latent diffusion model capable of generating realistic synthetic images with labels. The sampling time can be reduced to several tens of images per seconds. We demonstrate that models can be effectively trained using only a small set of synthetic images and evaluated on a large real test set. Our approach achieved rank in The First Dataset Distillation Challenge at ECCV 2024 on the CIFAR100 and TinyImageNet datasets.
Paper Structure (14 sections, 10 equations, 7 figures, 2 tables)

This paper contains 14 sections, 10 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Data Distillation
  5. Generative Diffusion Models and Risk Minimization
  6. Problem Formulation
  7. Method
  8. UViT Architecture
  9. Generative Diffusion Process
  10. Experiments
  11. Datasets
  12. Implement Details
  13. Results
  14. Conclusion

Figures (7)

  • Figure 1: An overview of dataset distillation task. The upper half of the image represents the typical matching methods and the lower half demonstrates how to generate synthetic images using a diffusion model.
  • Figure 2: The architecture applied in diffusion models. The architecture iteratively refines noise predictions using a combination of multi-layer networks and multi-head attention mechanisms.
  • Figure 3: The FID scores in every $1000$ iterations of training.
  • Figure 4: The Leaderboard of the Challenge. We are the rank $1$zheli team in Track $2$.
  • Figure 5: The generated images ($32\times 32$) by the diffusion model trained on CIFAR100 dataset. Each represents a class.
  • ...and 2 more figures

Theorems & Definitions (2)

  • definition thmcounterdefinition: Data Distillation
  • definition thmcounterdefinition: Optimal Data Distillation