A Recipe for Improved Certifiable Robustness

TL;DR

This work tackles certifiable robustness by focusing on Lipschitz-based certification and investigating a comprehensive design space for architecture, Lipschitz control, and data augmentation. It introduces the Cholesky-Orthogonalized Residual (CHORD) layers and dense residual blocks to efficiently expand capacity while maintaining tight Lipschitz bounds, complemented by an improved diffusion-based data augmentation pipeline. Empirical results show state-of-the-art deterministic verified robustness (VRA) across CIFAR-10/100 and Tiny ImageNet, and notable gains on ImageNet with generated data, narrowing the gap to probabilistic RS methods. The findings suggest that carefully engineered architectures and data pipelines can significantly boost deterministic certification, enabling more practical deployment of robust models.

Abstract

Recent studies have highlighted the potential of Lipschitz-based methods for training certifiably robust neural networks against adversarial attacks. A key challenge, supported both theoretically and empirically, is that robustness demands greater network capacity and more data than standard training. However, effectively adding capacity under stringent Lipschitz constraints has proven more difficult than it may seem, evident by the fact that state-of-the-art approach tend more towards \emph{underfitting} than overfitting. Moreover, we posit that a lack of careful exploration of the design space for Lipshitz-based approaches has left potential performance gains on the table. In this work, we provide a more comprehensive evaluation to better uncover the potential of Lipschitz-based certification methods. Using a combination of novel techniques, design optimizations, and synthesis of prior work, we are able to significantly improve the state-of-the-art VRA for deterministic certification on a variety of benchmark datasets, and over a range of perturbation sizes. Of particular note, we discover that the addition of large ``Cholesky-orthogonalized residual dense'' layers to the end of existing state-of-the-art Lipschitz-controlled ResNet architectures is especially effective for increasing network capacity and performance. Combined with filtered generative data augmentation, our final results further the state of the art deterministic VRA by up to 8.5 percentage points\footnote{Code is available at \url{https://github.com/hukkai/liresnet}}.

Figures (1)

• Figure 1: Certified accuracy (i.e., VRA) of our modified LiResNet architecture on CIFAR-10 using different Lipschitz control methods on the dense layers during training.