Robust Dataset Distillation by Matching Adversarial Trajectories

TL;DR

This work introduces Robust Dataset Distillation by Matching Adversarial Trajectories (MAT), a framework that embeds adversarial robustness directly into distilled data by aligning smoothed adversarial training trajectories with a teacher trajectory. MAT uses exponential moving average (EMA) smoothing to tame rapid weight changes during adversarial training, enabling effective trajectory matching and robust data synthesis. Experiments on CIFAR-10, CIFAR-100, and Tiny ImageNet show that models trained on MAT-distilled data achieve enhanced adversarial robustness with competitive clean accuracy, across multiple adversarial training regimes (PGD-AT, TRADES, MART) and architectures. The results establish robust dataset distillation as a viable, efficient path to reliable robust learning without the overhead of per-iteration adversarial training on large datasets.

Abstract

Dataset distillation synthesizes compact datasets that enable models to achieve performance comparable to training on the original large-scale datasets. However, existing distillation methods overlook the robustness of the model, resulting in models that are vulnerable to adversarial attacks when trained on distilled data. To address this limitation, we introduce the task of ``robust dataset distillation", a novel paradigm that embeds adversarial robustness into the synthetic datasets during the distillation process. We propose Matching Adversarial Trajectories (MAT), a method that integrates adversarial training into trajectory-based dataset distillation. MAT incorporates adversarial samples during trajectory generation to obtain robust training trajectories, which are then used to guide the distillation process. As experimentally demonstrated, even through natural training on our distilled dataset, models can achieve enhanced adversarial robustness while maintaining competitive accuracy compared to existing distillation methods. Our work highlights robust dataset distillation as a new and important research direction and provides a strong baseline for future research to bridge the gap between efficient training and adversarial robustness.

Figures (7)

• Figure 1: Overview of robust dataset distillation. The process begins with generating adversarial examples from the original dataset. These examples are then used to distill a robust dataset. Models trained on this distilled dataset through standard training naturally acquire adversarial robustness, eliminating the need for explicit adversarial training.
• Figure 2: Comparison of adversarial accuracy (%) on CIFAR-10 across different dataset distillation methods with natural training. We set IPC (images per class) to 10 and 50. Adversarial robustness was evaluated using PGD-10 attacks mkadry2017towards with $\epsilon$ = 4/255. As a comparison, we include the adversarial accuracy of a naturally trained model on the full dataset. DD refers to standard Dataset Distillation using MTT cazenavette2022dataset, while DD with AT uses unmodified adversarial training trajectories, and MAT (our method) employs smoothed adversarial training trajectories.
• Figure 3: The variance of weight during training. We compared the weight variation during training across three conditions: natural training, adversarial training, and adversarial training in our MAT (flat adversarial training).
• Figure 4: Workflow of trajectory generation in our method. Unlike previous approaches illustrated on the left, our method enhances model robustness via adversarial training and applies EMA to smooth the model's training trajectories.
• Figure 5: Impact of hyper-parameters EMA decay on model performance. Based on CIFAR-10, we set the EMA decay rate to 0.99, 0.999, and 0.9999 while keeping other hyper-parameters fixed and observed the effect on performance.