TAET: Two-Stage Adversarial Equalization Training on Long-Tailed Distributions

TL;DR

The paper tackles adversarial robustness under long-tailed distributions, where head classes dominate data yet tail classes remain underrepresented. It identifies limitations in the current state-of-the-art AT-BSL, notably weak tail-class robustness and robust overfitting, and introduces TAET, a two-stage training framework that first stabilizes natural accuracy with cross-entropy, then applies Hierarchical Adversarial Equalization Learning (HARL) to balance robustness across classes. HARL combines Balanced Cross-Class Loss, Hierarchical Deviation Loss, and Rare Class Emphasis Loss to reduce overfitting and boost performance on weak classes, while a newBalanced Robustness metric complements balanced accuracy for long-tail robustness evaluation. Empirical results on CIFAR-10-LT, CIFAR-100-LT, and DermaMNIST show TAET achieves superior balanced accuracy and robustness, and does so with improved memory and computational efficiency, making it practical for real-world long-tailed data settings.

Abstract

Adversarial robustness is a critical challenge in deploying deep neural networks for real-world applications. While adversarial training is a widely recognized defense strategy, most existing studies focus on balanced datasets, overlooking the prevalence of long-tailed distributions in real-world data, which significantly complicates robustness. This paper provides a comprehensive analysis of adversarial training under long-tailed distributions and identifies limitations in the current state-of-the-art method, AT-BSL, in achieving robust performance under such conditions. To address these challenges, we propose a novel training framework, TAET, which integrates an initial stabilization phase followed by a stratified equalization adversarial training phase. Additionally, prior work on long-tailed robustness has largely ignored the crucial evaluation metric of balanced accuracy. To bridge this gap, we introduce the concept of balanced robustness, a comprehensive metric tailored for assessing robustness under long-tailed distributions. Extensive experiments demonstrate that our method surpasses existing advanced defenses, achieving significant improvements in both memory and computational efficiency. This work represents a substantial advancement in addressing robustness challenges in real-world applications. Our code is available at: https://github.com/BuhuiOK/TAET-Two-Stage-Adversarial-Equalization-Training-on-Long-Tailed-Distributions.

Figures (13)

• Figure 1: Top: The distribution of accuracy and adversarial robustness across different classes in a long-tail distributions, with gray bars representing the sample counts of each class. AT-BSL (left) exhibits poorer performance on the tail classes (7, 8, 9) and Class 3. Bottom: The evaluation includes both balanced accuracy and robustness, comparing long-tail recognition methods, adversarial training, state-of-the-art defenses, and our proposed approach. The results demonstrate that our method outperforms the others in both balanced accuracy and performance on weak classes.
• Figure 2: Top: Accuracy progression during training with AT-BSL (left) and under PGD-20 attack (right). Bottom:Accuracy progression during training with our method (left) and under PGD-20 attack (right).
• Figure 3: AT
• Figure 4: TRADES
• Figure 5: AT-BSL