Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative Dataset Distillation using Min-Max Diffusion Model

Junqiao Fan, Yunjiao Zhou, Min Chang Jordan Ren, Jianfei Yang

TL;DR

The paper tackles generative dataset distillation under strict time constraints by leveraging a diffusion-based generator to create a surrogate dataset. It introduces two min-max losses to enforce representativeness and diversity, and proposes Diffusion Step Reduction to balance image quality with the number of samples within the given time. Empirical results show strong performance, including a second-place finish in the ECCV2024 First Dataset Distillation Challenge and favorable IPC trade-offs. The approach demonstrates that diffusion models, when coupled with targeted losses and step control, can produce compact yet effective surrogate datasets for training classifiers.

Abstract

In this paper, we address the problem of generative dataset distillation that utilizes generative models to synthesize images. The generator may produce any number of images under a preserved evaluation time. In this work, we leverage the popular diffusion model as the generator to compute a surrogate dataset, boosted by a min-max loss to control the dataset's diversity and representativeness during training. However, the diffusion model is time-consuming when generating images, as it requires an iterative generation process. We observe a critical trade-off between the number of image samples and the image quality controlled by the diffusion steps and propose Diffusion Step Reduction to achieve optimal performance. This paper details our comprehensive method and its performance. Our model achieved $2^{nd}$ place in the generative track of \href{https://www.dd-challenge.com/#/}{The First Dataset Distillation Challenge of ECCV2024}, demonstrating its superior performance.

Generative Dataset Distillation using Min-Max Diffusion Model

TL;DR

The paper tackles generative dataset distillation under strict time constraints by leveraging a diffusion-based generator to create a surrogate dataset. It introduces two min-max losses to enforce representativeness and diversity, and proposes Diffusion Step Reduction to balance image quality with the number of samples within the given time. Empirical results show strong performance, including a second-place finish in the ECCV2024 First Dataset Distillation Challenge and favorable IPC trade-offs. The approach demonstrates that diffusion models, when coupled with targeted losses and step control, can produce compact yet effective surrogate datasets for training classifiers.

Abstract

In this paper, we address the problem of generative dataset distillation that utilizes generative models to synthesize images. The generator may produce any number of images under a preserved evaluation time. In this work, we leverage the popular diffusion model as the generator to compute a surrogate dataset, boosted by a min-max loss to control the dataset's diversity and representativeness during training. However, the diffusion model is time-consuming when generating images, as it requires an iterative generation process. We observe a critical trade-off between the number of image samples and the image quality controlled by the diffusion steps and propose Diffusion Step Reduction to achieve optimal performance. This paper details our comprehensive method and its performance. Our model achieved place in the generative track of \href{https://www.dd-challenge.com/#/}{The First Dataset Distillation Challenge of ECCV2024}, demonstrating its superior performance.

Paper Structure

This paper contains 13 sections, 4 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Diffusion-based Dataset Distillation
  4. Min-max Training Loss
  5. Representativeness.
  6. Diversity.
  7. Diffusion Step Reduction
  8. Experiments
  9. Experiment Setup
  10. Implementation Details
  11. Overall Results
  12. Ablation Studies
  13. Conclusion

Figures (2)

  • Figure 1: During training, the diffusion model learns a dataset distribution by gradually adding Gaussian noise to images and reversing back. The model is trained by $\mathcal{L}_{train} = \mathcal{L}_{diff} + \lambda_r*\mathcal{L}_r + \lambda_d*\mathcal{L}_d$. During inference, the trained model utilizes 10 mins to generate samples for the surrogate dataset, where DSR is utilized to accelerate image generation.
  • Figure 2: Qualitative visualization of the surrogate dataset on ImageNet-tiny and CIFAR-100, where 10 and 30 diffusion iterations are performed to generate images.