Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Robust Fairness via Confusional Spectral Regularization

Gaojie Jin, Sihao Wu, Jiaxu Liu, Tianjin Huang, Ronghui Mu

TL;DR

This paper tackles robust fairness in adversarial robustness, where class-wise robust accuracy can be highly uneven. It develops a robust PAC-Bayesian bound for the worst-class robust error, showing the bound is governed by the spectral norm of the empirical robust confusion matrix plus a model/data-dependent term. To operationalize this, it proposes a differentiable spectral-regularization technique that targets the confusional spectral norm via a surrogate confusion matrix and an adversarial training objective, enabling gradient-based optimization. Extensive experiments on CIFAR-10/100 and Tiny-ImageNet—including fine-tuning, DDPM-augmented data, and full training—demonstrate improvements in worst-class robust accuracy and overall robustness while maintaining competitive average performance. This work provides a principled framework and practical algorithm for improving robust fairness in DNNs under adversarial perturbations, with implications for more reliable deployment in safety-critical settings.

Abstract

Recent research has highlighted a critical issue known as ``robust fairness", where robust accuracy varies significantly across different classes, undermining the reliability of deep neural networks (DNNs). A common approach to address this has been to dynamically reweight classes during training, giving more weight to those with lower empirical robust performance. However, we find there is a divergence of class-wise robust performance between training set and testing set, which limits the effectiveness of these explicit reweighting methods, indicating the need for a principled alternative. In this work, we derive a robust generalization bound for the worst-class robust error within the PAC-Bayesian framework, accounting for unknown data distributions. Our analysis shows that the worst-class robust error is influenced by two main factors: the spectral norm of the empirical robust confusion matrix and the information embedded in the model and training set. While the latter has been extensively studied, we propose a novel regularization technique targeting the spectral norm of the robust confusion matrix to improve worst-class robust accuracy and enhance robust fairness. We validate our approach through comprehensive experiments on various datasets and models, demonstrating its effectiveness in enhancing robust fairness.

Enhancing Robust Fairness via Confusional Spectral Regularization

TL;DR

This paper tackles robust fairness in adversarial robustness, where class-wise robust accuracy can be highly uneven. It develops a robust PAC-Bayesian bound for the worst-class robust error, showing the bound is governed by the spectral norm of the empirical robust confusion matrix plus a model/data-dependent term. To operationalize this, it proposes a differentiable spectral-regularization technique that targets the confusional spectral norm via a surrogate confusion matrix and an adversarial training objective, enabling gradient-based optimization. Extensive experiments on CIFAR-10/100 and Tiny-ImageNet—including fine-tuning, DDPM-augmented data, and full training—demonstrate improvements in worst-class robust accuracy and overall robustness while maintaining competitive average performance. This work provides a principled framework and practical algorithm for improving robust fairness in DNNs under adversarial perturbations, with implications for more reliable deployment in safety-critical settings.

Abstract

Recent research has highlighted a critical issue known as ``robust fairness", where robust accuracy varies significantly across different classes, undermining the reliability of deep neural networks (DNNs). A common approach to address this has been to dynamically reweight classes during training, giving more weight to those with lower empirical robust performance. However, we find there is a divergence of class-wise robust performance between training set and testing set, which limits the effectiveness of these explicit reweighting methods, indicating the need for a principled alternative. In this work, we derive a robust generalization bound for the worst-class robust error within the PAC-Bayesian framework, accounting for unknown data distributions. Our analysis shows that the worst-class robust error is influenced by two main factors: the spectral norm of the empirical robust confusion matrix and the information embedded in the model and training set. While the latter has been extensively studied, we propose a novel regularization technique targeting the spectral norm of the robust confusion matrix to improve worst-class robust accuracy and enhance robust fairness. We validate our approach through comprehensive experiments on various datasets and models, demonstrating its effectiveness in enhancing robust fairness.
Paper Structure (20 sections, 9 theorems, 48 equations, 4 figures, 11 tables)

This paper contains 20 sections, 9 theorems, 48 equations, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Worst-class robust error and confusional spectral norm
  4. Sketch of Proof
  5. Empirical results
  6. Spectral regularization of confusion matrix
  7. Experiments
  8. Conclusion
  9. Related work
  10. Proof of Lem. \ref{['lem:3.2']}
  11. Proof of Lem. \ref{['lem:3.3']}
  12. Proof for Lem. \ref{['lem:3.4']}
  13. More experimental results
  14. Sensitivity analysis for hyper-parameters
  15. Experiments on clean training
  16. ...and 5 more sections

Key Result

Theorem 2.1

Consider a training dataset $\mathcal{S}$ with $m$ samples drawn from a distribution $\mathcal{D}$ on $\mathcal{X}_B \times \mathcal{Y}$ with $\mathcal{Y} = \{1, \ldots, d_y\}$. Given a learning algorithm (e.g., a classifier) with prior and posterior distributions $P$ and $Q$ (i.e., $\mathbf{w}+\ul$ where $m_{min}$ represents the minimal number of examples from $\mathcal{S}$ which belong to the sa

Figures (4)

  • Figure 1: Illustration of the theoretical framework: worst-class robust generalization bound. Under this framework, a standard generalization bound over confusion matrix is extended to a robust generalization bound for the worst-class robust error.
  • Figure 2: Left: Class-wise AA (Auto Attack) accuracy of the adversarially trained WideResNet-28-10 model on CIFAR-10. The star points are the worst-class AA accuracy on training set and testing set. Right: The covariance (blue) and Kendall rank correlation (orange) of class-wise AA accuracy between training set and testing set for normal adversarially trained WRN-28-10 model, reweighting methods (FRL and FAAL) fine-tuned models, and our method fine-tuned model.
  • Figure 3: In both confusion matrices, the horizontal axis represents the true labels, while the vertical axis represents the predicted labels. The left figure shows the AA results of a WRN-34-10 model trained using the TRADES method on CIFAR-10, whereas the right figure demonstrates the AA results of a WRN-34-10 model trained using our method with $\gamma=0.1$.
  • Figure 4: We adversarially trained Preact-ResNet-18 models for standard TRADES (left) and our method with $\gamma=0.0$ based on TRADES (right) for 200 epochs using SGD with a momentum of 0.9, batch size of 256, weight decay of $5\times 10^{-4}$, and an initial learning rate of 0.1, which is reduced by a factor of 10 at the 100th and 150th epochs. Blue line represents training accuracy under PGD-10, while orange line represents testing accuracy under PGD-10.

Theorems & Definitions (20)

  • Remark 1
  • Theorem 2.1: morvant2012pac
  • Remark 2: Difference with previous work
  • Proposition 3.1
  • Remark 3
  • Lemma 3.2
  • Lemma 3.3
  • Lemma 3.4
  • Proof B.1
  • Proof B.2
  • ...and 10 more