Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MixedNUTS: Training-Free Accuracy-Robustness Balance via Nonlinearly Mixed Classifiers

Yatong Bai, Mo Zhou, Vishal M. Patel, Somayeh Sojoudi

TL;DR

MixedNUTS is proposed, a training-free method where the output logits of a robust classifier and a standard non-robust classifier are processed by nonlinear transformations with only three parameters, which are optimized through an efficient algorithm.

Abstract

Adversarial robustness often comes at the cost of degraded accuracy, impeding real-life applications of robust classification models. Training-based solutions for better trade-offs are limited by incompatibilities with already-trained high-performance large models, necessitating the exploration of training-free ensemble approaches. Observing that robust models are more confident in correct predictions than in incorrect ones on clean and adversarial data alike, we speculate amplifying this "benign confidence property" can reconcile accuracy and robustness in an ensemble setting. To achieve so, we propose "MixedNUTS", a training-free method where the output logits of a robust classifier and a standard non-robust classifier are processed by nonlinear transformations with only three parameters, which are optimized through an efficient algorithm. MixedNUTS then converts the transformed logits into probabilities and mixes them as the overall output. On CIFAR-10, CIFAR-100, and ImageNet datasets, experimental results with custom strong adaptive attacks demonstrate MixedNUTS's vastly improved accuracy and near-SOTA robustness -- it boosts CIFAR-100 clean accuracy by 7.86 points, sacrificing merely 0.87 points in robust accuracy.

MixedNUTS: Training-Free Accuracy-Robustness Balance via Nonlinearly Mixed Classifiers

TL;DR

MixedNUTS is proposed, a training-free method where the output logits of a robust classifier and a standard non-robust classifier are processed by nonlinear transformations with only three parameters, which are optimized through an efficient algorithm.

Abstract

Adversarial robustness often comes at the cost of degraded accuracy, impeding real-life applications of robust classification models. Training-based solutions for better trade-offs are limited by incompatibilities with already-trained high-performance large models, necessitating the exploration of training-free ensemble approaches. Observing that robust models are more confident in correct predictions than in incorrect ones on clean and adversarial data alike, we speculate amplifying this "benign confidence property" can reconcile accuracy and robustness in an ensemble setting. To achieve so, we propose "MixedNUTS", a training-free method where the output logits of a robust classifier and a standard non-robust classifier are processed by nonlinear transformations with only three parameters, which are optimized through an efficient algorithm. MixedNUTS then converts the transformed logits into probabilities and mixes them as the overall output. On CIFAR-10, CIFAR-100, and ImageNet datasets, experimental results with custom strong adaptive attacks demonstrate MixedNUTS's vastly improved accuracy and near-SOTA robustness -- it boosts CIFAR-100 clean accuracy by 7.86 points, sacrificing merely 0.87 points in robust accuracy.
Paper Structure (43 sections, 4 theorems, 19 equations, 9 figures, 11 tables, 1 algorithm)

This paper contains 43 sections, 4 theorems, 19 equations, 9 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Definitions and Notations
  4. Related Adversarial Attacks and Defenses
  5. Ensemble and Calibration
  6. Mixing Classifiers for Accuracy-Robust Trade-Off
  7. Base Classifier Confidence Modification
  8. Accurate Base Classifier Temperature Scaling
  9. MixedNUTS -- Nonlinearly Mixed Classifier
  10. Nonlinear Robust Base Classifier Transformation
  11. Parameterizing the Transformation M
  12. Efficient Algorithm for Optimizing s, p, c, and a
  13. Visualization of the Nonlinear Logit Transformation M(s,p,c)
  14. Experiments
  15. Main Experiment Results
  16. ...and 28 more sections

Key Result

Theorem 4.3

Suppose that ass:transf holds. Let $r_{ f^M_{\textrm{mix}}}$ and $r_{ h_{\textrm{rob}}}$ denote the robust accuracy of $f^M_{\textrm{mix}}(\cdot)$ and $h_{\textrm{rob}}(\cdot)$ respectively. If $\beta \geq r_{ f^M_{\textrm{mix}}}/r_{ h_{\textrm{rob}}}$, then a solution to eq:T_opt_2 is f

Figures (9)

  • Figure 1: Overview of the proposed MixedNUTS classifier. The nonlinear logit transformation, to be introduced in \ref{['sec:nlmc']}, significantly improves the accuracy-robustness balance while only introducing three parameters efficiently optimized with \ref{['alg:spca_opt']}.
  • Figure 2: MixedNUTS's accuracy-robustness balance compared to state-of-the-art models on RobustBench. MixedNUTS is more accurate on clean data than all standalone robust models. At the same time, MixedNUTS achieves the second-highest robustness among all models for CIFAR-100 and ImageNet, and is the third most robust for CIFAR-10.
  • Figure 3: The raw logits, the corresponding prediction probabilities, and the probabilities computed with the transformed logits. Our transformation augments the confidence margin difference between the two scenarios.
  • Figure 4: Probability trajectories on the probability simplex formed by temperature scaling, with or without the logit transformation. The transformation reduces confidence when classes compete.
  • Figure 5: MixedNUTS balances the robustness from its robust base classifier and the accuracy from its standard base classifier. The nonlinear logit transformation helps MixedNUTS achieve a much better accuracy-robust trade-off than a baseline mixed model without transformation. \ref{['sec:mixing_details']} reports the base model details and the optimal $s$, $p$, $c$, $\alpha$ values.
  • ...and 4 more figures

Theorems & Definitions (8)

  • Definition 2.1
  • Definition 2.2
  • Theorem 4.3
  • Theorem 4.4
  • Theorem 4.3 (restated)
  • proof
  • Theorem 4.4 (restated)
  • proof