Compositional Curvature Bounds for Deep Neural Networks

TL;DR

This work addresses adversarial vulnerability by introducing a curvature-based framework for neural networks, where the curvature constant $L_{Df}$ (the Jacobian’s Lipschitz constant) governs robustness beyond traditional Lipschitz bounds. It develops an analytical, scalable algorithm that propagates curvature bounds through model compositions and residual blocks, yielding differentiable curvature regularizers and anchored Lipschitz constants to tighten robustness certificates. The paper derives curvature-based robustness and attack certificates with closed-form radii, and furnishes a practical algorithm to bound the end-to-end Jacobian while enabling curvature-aware training. Empirical results on MNIST and CIFAR-10 demonstrate improved certified radii and reduced certification gaps, and the approach supports 1-Lipschitz architectures to further enhance robustness with practical training strategies.

Abstract

A key challenge that threatens the widespread use of neural networks in safety-critical applications is their vulnerability to adversarial attacks. In this paper, we study the second-order behavior of continuously differentiable deep neural networks, focusing on robustness against adversarial perturbations. First, we provide a theoretical analysis of robustness and attack certificates for deep classifiers by leveraging local gradients and upper bounds on the second derivative (curvature constant). Next, we introduce a novel algorithm to analytically compute provable upper bounds on the second derivative of neural networks. This algorithm leverages the compositional structure of the model to propagate the curvature bound layer-by-layer, giving rise to a scalable and modular approach. The proposed bound can serve as a differentiable regularizer to control the curvature of neural networks during training, thereby enhancing robustness. Finally, we demonstrate the efficacy of our method on classification tasks using the MNIST and CIFAR-10 datasets.

Figures (9)

• Figure 1: Depiction of anchored Lipschitz constants for $f(x) = \tanh(x)$. The anchored Lipschitz constant at $x=2$ is less than $0.582$, whereas the global Lipschitz constant is $1$.
• Figure 2: Certified ($\underline{\varepsilon}^*$) and attack ($\overline{\varepsilon}^*$) radii estimates. The green tangent circle denotes the certified radius $\varepsilon^*$.
• Figure 3: Certified radius comparison on a 6-layer neural network.
• Figure 4: Comparison of global Lipschitz and Curvature estimation against their anchored counterparts. The shaded areas denote the standard deviation over the whole dataset.
• Figure 5: Histogram of the certified attack radii for a 6-layer neural network trained via curvature regularization on CIFAR-10.

Theorems & Definitions (31)

• Definition 1.1: Anchored Lipschitz constant
• Lemma 1.2
• Proposition 2.1: Curvature-based certified radius
• Proposition 2.2
• Proposition 2.3
• Proposition 2.4: Curvature-based attack certificate
• Theorem 3.1: Compositional curvature estimation
• Theorem 3.2: Anchored compositional curvature estimation
• Remark 3.3
• Corollary 3.4