BEARD: Benchmarking the Adversarial Robustness for Dataset Distillation

TL;DR

BEARD is introduced, an open and unified benchmark designed to systematically assess the adversarial robustness of DD methods, including DM, IDM, and BACON, and introduces three key metrics: Robustness Ratio (RR), Attack Efficiency Ratio (AE), and Comprehensive Robustness-Efficiency Index (CREI).

Abstract

Dataset Distillation (DD) is an emerging technique that compresses large-scale datasets into significantly smaller synthesized datasets while preserving high test performance and enabling the efficient training of large models. However, current research primarily focuses on enhancing evaluation accuracy under limited compression ratios, often overlooking critical security concerns such as adversarial robustness. A key challenge in evaluating this robustness lies in the complex interactions between distillation methods, model architectures, and adversarial attack strategies, which complicate standardized assessments. To address this, we introduce BEARD, an open and unified benchmark designed to systematically assess the adversarial robustness of DD methods, including DM, IDM, and BACON. BEARD encompasses a variety of adversarial attacks (e.g., FGSM, PGD, C&W) on distilled datasets like CIFAR-10/100 and TinyImageNet. Utilizing an adversarial game framework, it introduces three key metrics: Robustness Ratio (RR), Attack Efficiency Ratio (AE), and Comprehensive Robustness-Efficiency Index (CREI). Our analysis includes unified benchmarks, various Images Per Class (IPC) settings, and the effects of adversarial training. Results are available on the BEARD Leaderboard, along with a library providing model and dataset pools to support reproducible research. Access the code at BEARD.

Figures (6)

• Figure 1: Illustration of evaluating adversarial robustness for dataset distillation: The process is divided into three stages: 1) Distillation stage, where diverse dataset distillation methods such as DC b1, DSA b2, and DM b3 generate distilled datasets. 2) Training stage, where models are trained on these distilled datasets. 3) Evaluating stage, where adversarial attacks (e.g., FGSM b6, PGD b7, and C&W b8) are applied to the test set of standard datasets like CIFAR-10/100 b13 and TinyImageNet b14, and model performance is evaluated both with and without adversarial attacks, summarized using specific metrics.
• Figure 2: Illustration of BEARD: We first obtain a distilled dataset pool from the source dataset using various dataset distillation methods, such as DC b1, DSA b2, DM b3, IDM b4, BACON b5, among others. Next, we train neural networks on these diverse distilled datasets to generate a collection of pretrained models, forming our model pool. Finally, we evaluate the adversarial robustness of the models in the model pool by applying a variety of adversarial attack methods, including FGSM b6, PGD b7, C&W b8, DeepFool b9, AutoAttack b10, and others.
• Figure 3: The top three entries on our CIFAR-10 leaderboard, with unified IPC settings, are available at https://beard-leaderboard.github.io/. The leaderboard utilizes metrics such as CREI, RRM, and AEM to assess robustness and attack efficiency. Additionally, it provides links to the code and distilled datasets for each entry, along with detailed information regarding authors, venues, and the last update.
• Figure 4: Performance of various dataset distillation methods under targeted and untargeted adversarial attacks on CIFAR-10, CIFAR-100, and TinyImageNet. The first row depicts targeted attacks with unified IPC settings, while the second row shows performance under untargeted attacks. Metrics used include Multi-Adversary Robustness Ratio (RRM), Multi-Adversary Attack Efficiency Ratio (AEM), and Comprehensive Robustness-Efficiency Index (CREI).
• Figure 5: CREI trends under targeted and untargeted attacks across three datasets: CIFAR-10, CIFAR-100, and TinyImageNet. The x-axis represents the number of IPC, while the y-axis displays CREI values. Six DD methods (DC, DSA, MTT, DM, IDM, BACON) are compared to full-size datasets at IPC-1, IPC-10, and IPC-50, highlighting their robustness and efficiency across various attacks.

Theorems & Definitions (11)

• Definition 1: Attacker Function
• Definition 2: Defender Function
• Definition 3: Attack Success Rate (ASR)
• Definition 4: Attack Success Time (AST)
• Definition 5: Adversarial Game Framework
• Remark 1
• Definition 6: Robustness Ratio
• Remark 2
• Definition 7: Attack Efficiency Ratio
• Definition 8: Comprehensive Robustness-Efficiency Index