DataDAM: Efficient Dataset Distillation with Attention Matching

TL;DR

DataDAM tackles the high cost of training large models by learning a compact, informative synthetic dataset through attention-guided distillation. It introduces Spatial Attention Matching (SAM) across multiple layers of randomly initialized networks and couples it with a last-layer feature distribution regularizer, formulating the objective as $\mathcal{L}_{\text{SAM}} + \lambda \mathcal{L}_{\text{MMD}}$. Across CIFAR-10/100, TinyImageNet, ImageNet-1K, and subsets, DataDAM achieves state-of-the-art results with substantial reductions in training time and memory, including up to $6.5\%$ gains on CIFAR-100 and $4.1\%$ on ImageNet-1K, while remaining robust to initialization and cross-architecture transfer. The distilled data also improves continual learning replay strategies and accelerates neural architecture search, demonstrating practical impact in memory efficiency and rapid model evaluation. While highly effective for CNNs, DataDAM’s current formulation limits cross-architecture generalization to Vision Transformers and requires re-optimization when the IPC changes, guiding future work toward broader applicability and efficiency.

Abstract

Researchers have long tried to minimize training costs in deep learning while maintaining strong generalization across diverse datasets. Emerging research on dataset distillation aims to reduce training costs by creating a small synthetic set that contains the information of a larger real dataset and ultimately achieves test accuracy equivalent to a model trained on the whole dataset. Unfortunately, the synthetic data generated by previous methods are not guaranteed to distribute and discriminate as well as the original training data, and they incur significant computational costs. Despite promising results, there still exists a significant performance gap between models trained on condensed synthetic sets and those trained on the whole dataset. In this paper, we address these challenges using efficient Dataset Distillation with Attention Matching (DataDAM), achieving state-of-the-art performance while reducing training costs. Specifically, we learn synthetic images by matching the spatial attention maps of real and synthetic data generated by different layers within a family of randomly initialized neural networks. Our method outperforms the prior methods on several datasets, including CIFAR10/100, TinyImageNet, ImageNet-1K, and subsets of ImageNet-1K across most of the settings, and achieves improvements of up to 6.5% and 4.1% on CIFAR100 and ImageNet-1K, respectively. We also show that our high-quality distilled images have practical benefits for downstream applications, such as continual learning and neural architecture search.

Figures (29)

• Figure 1: (a) Data distribution of the distilled images on the CIFAR10 dataset with 50 images per class (IPC50) for CAFE wang2022cafe and DataDAM. (b) Performance comparison with state-of-the-art methods on the CIFAR10 dataset for varying IPCs.
• Figure 2: (a) Illustration of the proposed DataDAM method. DataDAM includes a Spatial Attention Matching (SAM) module to capture the dataset's distribution and a complementary loss for matching the feature distributions in the last layer of the encoder network. (b) The internal architecture of the SAM module.
• Figure 3: Test accuracy evolution of synthetic image learning on CIFAR10 with IPC50 under three different initializations: Random, K-Center, and Gaussian noise.
• Figure 4: The effect of task balance $\lambda$ on the testing accuracy (%) for CIFAR10 dataset with IPC10 configuration.
• Figure 5: Distributions of synthetic images learned by four methods on CIFAR10 with IPC50. The stars represent the synthetic data dispersed amongst the original training dataset.