Understanding Certified Training with Interval Bound Propagation

TL;DR

This work investigates why interval bound propagation (IBP)–based certified training, despite using imprecise bounds, achieves strong robustness. It introduces propagation invariance and a global tightness metric $\tau$ to quantify how close IBP bounds are to the exact optimal bounds, deriving necessary and sufficient conditions for tight IBP bounds in deep linear networks and extending insights to ReLU nets. The authors prove that IBP training increases bound tightness and show via theory and experiments that network width plays a crucial role in achieving strong certified robustness, with wider networks outperforming deeper ones in this setting. The results illuminate the robustness–accuracy trade-off in certified training and suggest pathways to novel training methods that balance certifiability with standard accuracy by controlling propagation tightness.

Abstract

As robustness verification methods are becoming more precise, training certifiably robust neural networks is becoming ever more relevant. To this end, certified training methods compute and then optimize an upper bound on the worst-case loss over a robustness specification. Curiously, training methods based on the imprecise interval bound propagation (IBP) consistently outperform those leveraging more precise bounding methods. Still, we lack an understanding of the mechanisms making IBP so successful. In this work, we thoroughly investigate these mechanisms by leveraging a novel metric measuring the tightness of IBP bounds. We first show theoretically that, for deep linear models, tightness decreases with width and depth at initialization, but improves with IBP training, given sufficient network width. We, then, derive sufficient and necessary conditions on weight matrices for IBP bounds to become exact and demonstrate that these impose strong regularization, explaining the empirically observed trade-off between robustness and accuracy in certified training. Our extensive experimental evaluation validates our theoretical predictions for ReLU networks, including that wider networks improve performance, yielding state-of-the-art results. Interestingly, we observe that while all IBP-based training methods lead to high tightness, this is neither sufficient nor necessary to achieve high certifiable robustness. This hints at the existence of new training methods that do not induce the strong regularization required for tight IBP bounds, leading to improved robustness and standard accuracy.

Figures (16)

• Figure 1: Comparison of exact (), optimal box (), and IBP () propagation through a one layer network. We show the concrete points maximizing the logit difference $y_2-y_1$ as a black $\times$ and the corresponding relaxation as a red $\times$.
• Figure 2: Mean relative error between local tightness (\ref{['def:relu_tightness']}) and true tightness computed with MILP for a CNN3 trained with PGD or IBP at $\epsilon=0.05$ on MNIST.
• Figure 3: Dependence of tightness at initialization on width (left) and depth (right) for a CNN7 and CIFAR-10.
• Figure 4: Box reconstruction error over bottleneck width $w$.
• Figure 5: Effect of network depth (top) and width (bottom) on tightness and training set IBP -certified accuracy.

Theorems & Definitions (34)

• Lemma 3.0
• Theorem 3.1: Box Propagation
• Lemma 3.1: Propagation Invariance
• Theorem 3.2: Non-Propagation Invariance
• Definition 3.3
• Definition 3.4
• Lemma 3.4: Initialization Tightness w.r.t. Width
• Corollary 3.4: Initialization Tightness w.r.t. Depth
• Theorem 3.5: IBP Training Increases Tightness
• Theorem 3.6: Box Reconstruction Error