Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Video Dataset Condensation with Diffusion Models

Zhe Li, Hadrien Reynaud, Mischa Dombrowski, Sarah Cechnicka, Franciskus Xaverius Erick, Bernhard Kainz

TL;DR

The paper tackles the high cost of video dataset growth by generating a large pool of synthetic videos with a class-conditional diffusion model and then distilling a compact, representative subset. It introduces VST-UNet, a 4D U-Net-based video selector that optimizes for diversity and representativeness, and TAC-DT, a training-free clustering method using VideoMAE embeddings and BIRCH clustering. Across four benchmarks, the approach yields significant improvements over state-of-the-art methods, with up to 10.61 percentage points gains and the ability to match full real-dataset performance on MiniUCF at higher videos-per-class. The framework offers a scalable, efficient solution for video dataset distillation with practical impact for resource-constrained training and deployment.

Abstract

In recent years, the rapid expansion of dataset sizes and the increasing complexity of deep learning models have significantly escalated the demand for computational resources, both for data storage and model training. Dataset distillation has emerged as a promising solution to address this challenge by generating a compact synthetic dataset that retains the essential information from a large real dataset. However, existing methods often suffer from limited performance, particularly in the video domain. In this paper, we focus on video dataset distillation. We begin by employing a video diffusion model to generate synthetic videos. Since the videos are generated only once, this significantly reduces computational costs. Next, we introduce the Video Spatio-Temporal U-Net (VST-UNet), a model designed to select a diverse and informative subset of videos that effectively captures the characteristics of the original dataset. To further optimize computational efficiency, we explore a training-free clustering algorithm, Temporal-Aware Cluster-based Distillation (TAC-DT), to select representative videos without requiring additional training overhead. We validate the effectiveness of our approach through extensive experiments on four benchmark datasets, demonstrating performance improvements of up to \(10.61\%\) over the state-of-the-art. Our method consistently outperforms existing approaches across all datasets, establishing a new benchmark for video dataset distillation.

Video Dataset Condensation with Diffusion Models

TL;DR

The paper tackles the high cost of video dataset growth by generating a large pool of synthetic videos with a class-conditional diffusion model and then distilling a compact, representative subset. It introduces VST-UNet, a 4D U-Net-based video selector that optimizes for diversity and representativeness, and TAC-DT, a training-free clustering method using VideoMAE embeddings and BIRCH clustering. Across four benchmarks, the approach yields significant improvements over state-of-the-art methods, with up to 10.61 percentage points gains and the ability to match full real-dataset performance on MiniUCF at higher videos-per-class. The framework offers a scalable, efficient solution for video dataset distillation with practical impact for resource-constrained training and deployment.

Abstract

In recent years, the rapid expansion of dataset sizes and the increasing complexity of deep learning models have significantly escalated the demand for computational resources, both for data storage and model training. Dataset distillation has emerged as a promising solution to address this challenge by generating a compact synthetic dataset that retains the essential information from a large real dataset. However, existing methods often suffer from limited performance, particularly in the video domain. In this paper, we focus on video dataset distillation. We begin by employing a video diffusion model to generate synthetic videos. Since the videos are generated only once, this significantly reduces computational costs. Next, we introduce the Video Spatio-Temporal U-Net (VST-UNet), a model designed to select a diverse and informative subset of videos that effectively captures the characteristics of the original dataset. To further optimize computational efficiency, we explore a training-free clustering algorithm, Temporal-Aware Cluster-based Distillation (TAC-DT), to select representative videos without requiring additional training overhead. We validate the effectiveness of our approach through extensive experiments on four benchmark datasets, demonstrating performance improvements of up to over the state-of-the-art. Our method consistently outperforms existing approaches across all datasets, establishing a new benchmark for video dataset distillation.
Paper Structure (11 sections, 3 equations, 2 figures, 7 tables, 1 algorithm)

This paper contains 11 sections, 3 equations, 2 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. VST-UNet
  5. TAC-DT
  6. Experiments
  7. Comparison with the state-of-the-art
  8. Ablation study
  9. Conclusion
  10. Future Work.
  11. Acknowledgments:

Figures (2)

  • Figure 1: Overview of Our Framework. The left illustrates the video selection process. On the right, VST-UNet (top) extracts video latent features via an Autoencoder for selection, while TAC-DT (bottom) selects representative videos without training using VideoMAE embeddings wang2023videomae.
  • Figure 2: The $16$ consecutive frames from a synthetic video of the class horserace in the UCF101 dataset.