CTBENCH: A Library and Benchmark for Certified Training

TL;DR

CTBench addresses the lack of fair, reproducible comparisons in certified training by providing a unified library and benchmark for deterministic $L_inity$-norm robustness. It integrates multiple core certification algorithms (e.g., IBP, CROWN-IBP, SABR, TAPS, STAPS, MTL-IBP) within standardized training schedules and hyperparameter tuning to enable fair, apples-to-apples evaluation. The study shows that CTBench achieves new state-of-the-art certified accuracy across several datasets and highlights that the purported advantages of recent methods often shrink when baselines are fairly tuned; it also yields actionable insights into loss fragmentation, shared mistakes, model utilization, regularization, and OOD generalization. By providing reproducible checkpoints and a comprehensive analysis, CTBench serves as a practical testbed and a foundation for future research in certified training and its applications.

Abstract

Training certifiably robust neural networks is an important but challenging task. While many algorithms for (deterministic) certified training have been proposed, they are often evaluated on different training schedules, certification methods, and systematically under-tuned hyperparameters, making it difficult to compare their performance. To address this challenge, we introduce CTBench, a unified library and a high-quality benchmark for certified training that evaluates all algorithms under fair settings and systematically tuned hyperparameters. We show that (1) almost all algorithms in CTBench surpass the corresponding reported performance in literature in the magnitude of algorithmic improvements, thus establishing new state-of-the-art, and (2) the claimed advantage of recent algorithms drops significantly when we enhance the outdated baselines with a fair training schedule, a fair certification method and well-tuned hyperparameters. Based on CTBench, we provide new insights into the current state of certified training, including (1) certified models have less fragmented loss surface, (2) certified models share many mistakes, (3) certified models have more sparse activations, (4) reducing regularization cleverly is crucial for certified training especially for large radii and (5) certified training has the potential to improve out-of-distribution generalization. We are confident that CTBench will serve as a benchmark and testbed for future research in certified training.

Figures (11)

• Figure 1: Reduction in certified error on MNIST$\epsilon=0.3$ (lower is better).
• Figure 2: Conceptual overview of core algorithms built into CTBench.
• Figure 3: Ratio of unstable neurons for models trained on MNIST with different methods and $\epsilon$.
• Figure 4: Model utilization for models trained on MNIST with different methods and $\epsilon$. We note that standard training has 42.99% utilization.
• Figure 5: Certified accuracy vs. propagation tightness for models trained on MNIST and CIFAR-10.