Towards Universal Certified Robustness with Multi-Norm Training

TL;DR

This work addresses the gap in certified robustness across multiple perturbation norms by introducing CURE, a deterministic multi-norm certified training framework designed to achieve union and universal robustness. It formalizes the l_q–l_r trade-off, develops bound-alignment and natural-training integration techniques, and proposes several training schemes (CURE-Joint, CURE-Max, CURE-Random) plus bound-alignment and quick certified fine-tuning to leverage pre-trained single-norm models. Empirical results on MNIST, CIFAR-10, and TinyImagenet show substantial gains in union robustness (e.g., up to 32.0% on MNIST, 25.8% on CIFAR-10, 10.6% on TinyImagenet) and improved robustness to unseen geometric/patch perturbations. These findings advance toward universal certified robustness by enabling efficient multi-norm training and effective fine-tuning of pre-trained models.

Abstract

Existing certified training methods can only train models to be robust against a certain perturbation type (e.g. $l_\infty$ or $l_2$). However, an $l_\infty$ certifiably robust model may not be certifiably robust against $l_2$ perturbation (and vice versa) and also has low robustness against other perturbations (e.g. geometric and patch transformation). By constructing a theoretical framework to analyze and mitigate the tradeoff, we propose the first multi-norm certified training framework \textbf{CURE}, consisting of several multi-norm certified training methods, to attain better \emph{union robustness} when training from scratch or fine-tuning a pre-trained certified model. Inspired by our theoretical findings, we devise bound alignment and connect natural training with certified training for better union robustness. Compared with SOTA-certified training, \textbf{CURE} improves union robustness to $32.0\%$ on MNIST, $25.8\%$ on CIFAR-10, and $10.6\%$ on TinyImagenet across different epsilon values. It leads to better generalization on a diverse set of challenging unseen geometric and patch perturbations to $6.8\%$ and $16.0\%$ on CIFAR-10. Overall, our contributions pave a path towards \textit{universal certified robustness}.

Figures (8)

• Figure 1: (a) $l_\infty - l_2$ tradeoff: an $l_\infty$ certified robust model may lack $l_2$ certified robustness and vice versa. CURE-Scratch (yellow) and CURE-Finetune (green) improve union robustness significantly. (b) We align the output bound differences for $l_q, l_r$ perturbations on the correctly certified $l_q$ subset $\gamma$ to mitigate $l_q - l_r$ tradeoff for better union robustness.
• Figure 2: $l_q-l_r$ trade-off visualization. The large rectangle represents all input image instances. Blue and Purple points are instances belonging to $\mathcal{R}_q$ and $\mathcal{R}_r$, respectively.
• Figure 3: Comparison on CURE against geometric transformations for MNIST $(\epsilon_1=2.0, \epsilon_2=1.0, \epsilon_\infty=0.3)$ experiment. We denote R$(\varphi)$ a rotation of $\pm \varphi$ degrees; Tu$(\Delta u)$ and Tv$(\Delta v)$ a translation of $\pm \Delta u$ pixels horizontally and $\pm \Delta v$ pixels vertically, respectively; Sc$(\lambda)$ a scaling of $\pm \lambda \%$; Sh$(\gamma)$ a shearing of $\pm \gamma \%$; C$(\alpha)$ a contrast change of $\pm \alpha \%$; and B$(\beta)$ a brightness change of $\pm \beta$. CURE improves the geometric certified robustness compared with single norm training. Also, CURE-Scratch achieves the best average geometric transformation robustness.
• Figure 4: CURE-Max and CURE-Scratch bound difference visualization.
• Figure 5: Alabtion studies on $\lambda_2$, $\eta$ and $\beta$ hyper-parameters.

Theorems & Definitions (4)

• Lemma 4.1
• Theorem 4.2
• Definition 4.3: Correctly Certified $l_r$ Subset
• proof