Tuning for Two Adversaries: Enhancing the Robustness Against Transfer and Query-Based Attacks using Hyperparameter Tuning

TL;DR

This work analyzes how training hyperparameters (learning rate, weight decay, momentum, batch size) shape robustness to transfer-based and query-based black-box attacks across centralized, ensemble, and distributed settings. It combines theory linking Hessian-based input-space smoothness and gradient similarity with extensive CIFAR-10 and ImageNet experiments, revealing a dichotomy: smaller learning rates boost transfer robustness while larger learning rates improve query robustness. The authors use NSGA-II to balance both attack types, showing distributed models achieve strong, dual-attack robustness with favorable efficiency. Overall, the paper provides practical hyperparameter tuning strategies that outperform some state-of-the-art defenses in robustness and efficiency.

Abstract

In this paper, we present the first detailed analysis of how training hyperparameters -- such as learning rate, weight decay, momentum, and batch size -- influence robustness against both transfer-based and query-based attacks. Supported by theory and experiments, our study spans a variety of practical deployment settings, including centralized training, ensemble learning, and distributed training. We uncover a striking dichotomy: for transfer-based attacks, decreasing the learning rate significantly enhances robustness by up to $64\%$. In contrast, for query-based attacks, increasing the learning rate consistently leads to improved robustness by up to $28\%$ across various settings and data distributions. Leveraging these findings, we explore -- for the first time -- the training hyperparameter space to jointly enhance robustness against both transfer-based and query-based attacks. Our results reveal that distributed models benefit the most from hyperparameter tuning, achieving a remarkable tradeoff by simultaneously mitigating both attack types more effectively than other training setups.

Figures (4)

• Figure 1: Impact of sharpness in parameter space (due to change of the learning rate hyperparameter) and smoothness in input space (left) and gradient similarity (right).
• Figure 2: Tension between adversarial examples (${ \hbox{$\m@th\blacktriangle$} }$) for transfer-based ($x'_T$) and query-based attacks ($x'_Q$) on a smooth (left) and a less smooth (right) model. The solid line and dotted line represent the decision boundaries for the target and surrogate model, respectively. The dashed line represents the $\varepsilon$ constraint around a benign sample ($x^*_{T/Q}$ / ).
• Figure 3: Robust accuracy for all hyperparameters $\mathcal{H}=(\eta, \lambda, \mu, B)$ for transfer-based ($\mathcal{A}_T$) and query-based attackers ($\mathcal{A}_Q$). Our results are averaged over all nodes for deep ensembles (solid-line) and distributed ML (dashed-line) on CIFAR-10. The blue line shows $\mathsf{RA}$, while the orange line shows $\mathsf{CA}$. Each column varies the specified hyperparameter, while fixing all others.
• Figure 4: Pareto frontier of the NSGA-II search across all ML instantiations and hyperparameters.

Theorems & Definitions (4)

• Definition 1: Stochastic Gradient Descent
• Definition 2: Model Smoothness
• Definition 3: Gradient Similarity
• Proposition 1: Less smooth models exhibit high robustness against transfer-based attacks