Reconcile Certified Robustness and Accuracy for DNN-based Smoothed Majority Vote Classifier

TL;DR

This work bridges generalization and certified robustness for DNNs by developing a margin-based PAC-Bayesian bound with a certified robust radius for smoothed Majority Vote classifiers. The key theoretical contributions include Lemma main1, Theorem main2, and Theorem main3, which collectively tie the generalization bound and the certified radius to the weight spectral norms, suggesting spectral-regularization as an effective training objective. A cheap spectral-regularization strategy is proposed, based on the output correlation under spherical Gaussian smoothing and Gershgorin circle considerations, implemented as an additional regularizer in smooth training. Empirical results across MNIST, Fashion-MNIST, CIFAR-10, and ImageNet show that reducing weight spectral norms via the proposed regularizer yields larger certified radii and improved certified robustness with modest overhead and manageable accuracy trade-offs. Overall, the paper provides both theoretical and practical advances toward jointly improving accuracy and verifiable robustness for smoothed ensemble classifiers.

Abstract

Within the PAC-Bayesian framework, the Gibbs classifier (defined on a posterior $Q$) and the corresponding $Q$-weighted majority vote classifier are commonly used to analyze the generalization performance. However, there exists a notable lack in theoretical research exploring the certified robustness of majority vote classifier and its interplay with generalization. In this study, we develop a generalization error bound that possesses a certified robust radius for the smoothed majority vote classifier (i.e., the $Q$-weighted majority vote classifier with smoothed inputs); In other words, the generalization bound holds under any data perturbation within the certified robust radius. As a byproduct, we find that the underpinnings of both the generalization bound and the certified robust radius draw, in part, upon weight spectral norm, which thereby inspires the adoption of spectral regularization in smooth training to boost certified robustness. Utilizing the dimension-independent property of spherical Gaussian inputs in smooth training, we propose a novel and inexpensive spectral regularizer to enhance the smoothed majority vote classifier. In addition to the theoretical contribution, a set of empirical results is provided to substantiate the effectiveness of our proposed method.

Figures (3)

• Figure 1: Illustration of the theoretical framework: perturbation bound for smoothed majority vote classifiers. Under this framework, a standard generalization bound is extended to a smoothed generalization bound with a certified robust radius.
• Figure 2: We train the base classifier for the Majority Vote MLPs on MNIST and FashionMNIST with $\alpha\in \{0.0, 0.1, 0.2, 0.3, 0.4, 0.5\}$, respectively. Left: the spectral norm of the whole weight matrix, i.e., $\|\mathbf{W}\|_2$ where $\mathbf{W}=\mathbf{W}_n \mathbf{W}_{n-1}\cdots\mathbf{W}_1$, with respect to $\alpha$. Middle: the product of the spectral norms of the weight matrix, i.e., $\prod_i \|\mathbf{W}_i\|_2$, with respect to $\alpha$. Right: the cosine similarity matrix of row vectors of the whole weight matrix $\mathbf{W}$.
• Figure 3: Experiments on the Majority Vote classifiers, for MLP on MNIST, MLP on FashionMNIST, ResNet110 on CIFAR-10, and ResNet50 on ImageNet, respectively. We certify the full MNIST, FashionMNIST, CIFAR-10 test sets and a subsample of 1000 examples from the ImageNet test set. Top: $\alpha=0.1$; Bottom: $\alpha=0.2$.

Theorems & Definitions (15)

• Lemma 2.1: mcallester2003simplifiedneyshabur2017pac
• Remark 1
• Remark 2
• Lemma 3.1
• Theorem 3.2
• Theorem 3.3
• Remark 3
• Proof 2.1
• Proof 2.2
• Proof 2.3