Fast and Flexible Robustness Certificates for Semantic Segmentation

TL;DR

This paper extends robustness certification to semantic segmentation by introducing Lipschitz-by-design networks and a fast, certificate-based framework. By converting input perturbations into bounded output changes, it derives tractable, GPU-friendly certificates for metrics such as Pixel Accuracy, FNR, and stability, and demonstrates real-time certifiability on Cityscapes. Empirical results show LipNet achieves competitive certification performance with orders-of-magnitude faster inference than randomized smoothing, albeit with a trade-off in raw segmentation accuracy. The work also provides a safety-critical use-case and a public codebase to train and certify Lipschitz-constrained segmentation models, highlighting the practical impact of real-time, certifiable robustness in safety-sensitive domains.

Abstract

Deep Neural Networks are vulnerable to small perturbations that can drastically alter their predictions for perceptually unchanged inputs. The literature on adversarially robust Deep Learning attempts to either enhance the robustness of neural networks (e.g, via adversarial training) or to certify their decisions up to a given robustness level (e.g, by using randomized smoothing, formal methods or Lipschitz bounds). These studies mostly focus on classification tasks and few efficient certification procedures currently exist for semantic segmentation. In this work, we introduce a new class of certifiably robust Semantic Segmentation networks with built-in Lipschitz constraints that are efficiently trainable and achieve competitive pixel accuracy on challenging datasets such as Cityscapes. Additionally, we provide a novel framework that generalizes robustness certificates for semantic segmentation tasks, where we showcase the flexibility and computational efficiency of using Lipschitz networks. Our approach unlocks real-time compatible certifiably robust semantic segmentation for the first time. Moreover, it allows the computation of worst-case performance under $\ell_2$ attacks of radius $ε$ across a wide range of performance measures. Crucially, we benchmark the runtime of our certification process and find our approach to be around 600 times faster than randomized smoothing methods at inference with comparable certificates on an NVIDIA A100 GPU. Finally, we evaluate the tightness of our worstcase certificates against state-of-the-art adversarial attacks to further validate the performance of our method.

Figures (8)

• Figure 1: (Left) The $\epsilon$ budget required to attack dense segmentations to make all but $N_\mathrm{min}$ pixels change. (Right) We display only the groups of predictions where $\epsilon \geq 0.1$, non-robust pixel groups are in white.
• Figure 2: We evaluate the certified pixel accuracy of SegCertify on the same $L$-Lipschitz neural network that performs Semantic Segmentation on the Oxford-IIIT Pet dataset parkhi2012cats. For SegCertify we carefully select $\sigma$ from $\{0.035, 0.05, 0.08, 0.1, 0.15, 0.2, 0.3 \}$ for each number of MC samples. Here, we certify the pixel accuracy against $\epsilon=0.1$ in $\ell_2$ norm. We pick $\alpha=0.01$ as the failure probability for $\mathrm{CRPA}_\mathrm{RS}$.
• Figure 3: Here, we attack 500 images from the Oxford-IIIT Pet dataset, for each different adversarial budget, we plot the CRPA and the actual empirical pixel accuracy under attack.
• Figure 4: Visualization of test set segmentation results using our Lipschitz constrained neural networks trained using the cosine similarity.
• Figure 5: Lipschitz neural networks allow for explicit control of their position on the robustness-accuracy tradeoff bethune2022pay, as seen on this small network trained on the Cats & Dogs dataset. This property remains valid in the context of semantic segmentation.

Theorems & Definitions (5)

• Definition 1: Robustness radius - classification
• Proposition 1: Classification robustness bound
• Definition 2: Worst-case performance
• Definition 3: Generalized robustness radius
• proof