Boosting Adversarial Training via Fisher-Rao Norm-based Regularization
Xiangyu Yin, Wenjie Ruan
TL;DR
This work addresses the persistent degradation of standard generalization in adversarial training by reframing model complexity through the geometric Fisher-Rao norm and tying it to the Cross-Entropy loss–based Rademacher complexity for ReLU networks. It identifies a logit-centered complexity variable $\Gamma_{ce}$ that captures how width and training objectives affect the generalization gap between adversarially trained and standard models, with epoch-dependent behavior. Building on these insights, the authors propose LOAT, a lightweight, epoch-aware regularization framework that combines standard logit-oriented penalties with adaptive adversarial logit pairing to reduce the CE generalization gap while preserving robustness. Extensive experiments across PGD-AT, TRADES, MART, and DM-AT on CIFAR-10 (and augmented datasets) show LOAT consistently improves clean accuracy and adversarial robustness with minimal overhead, demonstrating practical impact for improving robustness-generalization trade-offs in modern adversarial training pipelines.
Abstract
Adversarial training is extensively utilized to improve the adversarial robustness of deep neural networks. Yet, mitigating the degradation of standard generalization performance in adversarial-trained models remains an open problem. This paper attempts to resolve this issue through the lens of model complexity. First, We leverage the Fisher-Rao norm, a geometrically invariant metric for model complexity, to establish the non-trivial bounds of the Cross-Entropy Loss-based Rademacher complexity for a ReLU-activated Multi-Layer Perceptron. Then we generalize a complexity-related variable, which is sensitive to the changes in model width and the trade-off factors in adversarial training. Moreover, intensive empirical evidence validates that this variable highly correlates with the generalization gap of Cross-Entropy loss between adversarial-trained and standard-trained models, especially during the initial and final phases of the training process. Building upon this observation, we propose a novel regularization framework, called Logit-Oriented Adversarial Training (LOAT), which can mitigate the trade-off between robustness and accuracy while imposing only a negligible increase in computational overhead. Our extensive experiments demonstrate that the proposed regularization strategy can boost the performance of the prevalent adversarial training algorithms, including PGD-AT, TRADES, TRADES (LSE), MART, and DM-AT, across various network architectures. Our code will be available at https://github.com/TrustAI/LOAT.