Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DD-RobustBench: An Adversarial Robustness Benchmark for Dataset Distillation

Yifan Wu, Jiawei Du, Ping Liu, Yuewei Lin, Wei Xu, Wenqing Cheng

TL;DR

DD-RobustBench presents the first comprehensive benchmark for evaluating the adversarial robustness of distilled datasets across diverse distillation methods, attacks, and dataset scales, including large-scale ImageNet-1K subsets. It demonstrates that distilled data can offer superior robustness under many conditions, though robustness interacts with the compression ratio IPC and distillation components such as data augmentation and downsampling. The work provides a unified evaluation pipeline, critical insights into how different distillation elements affect robustness, and practical guidance for deploying and extending the benchmark. The benchmark thus offers a valuable tool for advancing robust dataset distillation in real-world applications.

Abstract

Dataset distillation is an advanced technique aimed at compressing datasets into significantly smaller counterparts, while preserving formidable training performance. Significant efforts have been devoted to promote evaluation accuracy under limited compression ratio while overlooked the robustness of distilled dataset. In this work, we introduce a comprehensive benchmark that, to the best of our knowledge, is the most extensive to date for evaluating the adversarial robustness of distilled datasets in a unified way. Our benchmark significantly expands upon prior efforts by incorporating a wider range of dataset distillation methods, including the latest advancements such as TESLA and SRe2L, a diverse array of adversarial attack methods, and evaluations across a broader and more extensive collection of datasets such as ImageNet-1K. Moreover, we assessed the robustness of these distilled datasets against representative adversarial attack algorithms like PGD and AutoAttack, while exploring their resilience from a frequency perspective. We also discovered that incorporating distilled data into the training batches of the original dataset can yield to improvement of robustness.

DD-RobustBench: An Adversarial Robustness Benchmark for Dataset Distillation

TL;DR

DD-RobustBench presents the first comprehensive benchmark for evaluating the adversarial robustness of distilled datasets across diverse distillation methods, attacks, and dataset scales, including large-scale ImageNet-1K subsets. It demonstrates that distilled data can offer superior robustness under many conditions, though robustness interacts with the compression ratio IPC and distillation components such as data augmentation and downsampling. The work provides a unified evaluation pipeline, critical insights into how different distillation elements affect robustness, and practical guidance for deploying and extending the benchmark. The benchmark thus offers a valuable tool for advancing robust dataset distillation in real-world applications.

Abstract

Dataset distillation is an advanced technique aimed at compressing datasets into significantly smaller counterparts, while preserving formidable training performance. Significant efforts have been devoted to promote evaluation accuracy under limited compression ratio while overlooked the robustness of distilled dataset. In this work, we introduce a comprehensive benchmark that, to the best of our knowledge, is the most extensive to date for evaluating the adversarial robustness of distilled datasets in a unified way. Our benchmark significantly expands upon prior efforts by incorporating a wider range of dataset distillation methods, including the latest advancements such as TESLA and SRe2L, a diverse array of adversarial attack methods, and evaluations across a broader and more extensive collection of datasets such as ImageNet-1K. Moreover, we assessed the robustness of these distilled datasets against representative adversarial attack algorithms like PGD and AutoAttack, while exploring their resilience from a frequency perspective. We also discovered that incorporating distilled data into the training batches of the original dataset can yield to improvement of robustness.
Paper Structure (23 sections, 1 equation, 2 figures, 13 tables)

This paper contains 23 sections, 1 equation, 2 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dataset Distillation
  4. Adversarial Robustness
  5. Benchmarks on Dataset Distillation
  6. Our Investigations
  7. Experimental Overview
  8. Experimental Setup
  9. Datasets
  10. Distillation Methods
  11. Adversarial Attack
  12. Network Architecture
  13. Implement Details
  14. Distillation Stage
  15. Training Stage
  16. ...and 8 more sections

Figures (2)

  • Figure 1: Illustration of our robustness evaluation pipeline. We first utilize dataset distillation for condensing the train set. Subsequently, the distilled data is used to train a random initialized network from scratch. The trained network is then evaluated on the perturbed original test set which is manipulated by adversarial attacks.
  • Figure 2: Drop rates after PGD attack. The dashed lines represent the $DR$ based on the original dataset. Higher $DR$ means relatively worse robustness.