Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Better Certified Models from Empirically-Robust Teachers

Alessandro De Palma

TL;DR

This work tackles the gap between empirical adversarial robustness and formal certifiability by distilling knowledge from empirically-robust teachers into certifiably robust student models. It introduces CC-Dist, a framework that couples a versatile feature-space distillation loss with an expressive certified training loss (CC-IBP), enabling the transfer of empirical robustness while preserving verifiability. Across CIFAR-10, TinyImageNet, and downscaled ImageNet, CC-Dist achieves new state-of-the-art certifiably-robust results for ReLU networks and improves both standard and certified accuracy relative to prior expressivity-based methods. The findings demonstrate that leveraging empirically-robust teachers can meaningfully enhance deterministic certificates, with applicability to diverse certified-training schemes and architectures.

Abstract

Adversarial training attains strong empirical robustness to specific adversarial attacks by training on concrete adversarial perturbations, but it produces neural networks that are not amenable to strong robustness certificates through neural network verification. On the other hand, earlier certified training schemes directly train on bounds from network relaxations to obtain models that are certifiably robust, but display sub-par standard performance. Recent work has shown that state-of-the-art trade-offs between certified robustness and standard performance can be obtained through a family of losses combining adversarial outputs and neural network bounds. Nevertheless, differently from empirical robustness, verifiability still comes at a significant cost in standard performance. In this work, we propose to leverage empirically-robust teachers to improve the performance of certifiably-robust models through knowledge distillation. Using a versatile feature-space distillation objective, we show that distillation from adversarially-trained teachers consistently improves on the state-of-the-art in certified training for ReLU networks across a series of robust computer vision benchmarks.

Learning Better Certified Models from Empirically-Robust Teachers

TL;DR

This work tackles the gap between empirical adversarial robustness and formal certifiability by distilling knowledge from empirically-robust teachers into certifiably robust student models. It introduces CC-Dist, a framework that couples a versatile feature-space distillation loss with an expressive certified training loss (CC-IBP), enabling the transfer of empirical robustness while preserving verifiability. Across CIFAR-10, TinyImageNet, and downscaled ImageNet, CC-Dist achieves new state-of-the-art certifiably-robust results for ReLU networks and improves both standard and certified accuracy relative to prior expressivity-based methods. The findings demonstrate that leveraging empirically-robust teachers can meaningfully enhance deterministic certificates, with applicability to diverse certified-training schemes and architectures.

Abstract

Adversarial training attains strong empirical robustness to specific adversarial attacks by training on concrete adversarial perturbations, but it produces neural networks that are not amenable to strong robustness certificates through neural network verification. On the other hand, earlier certified training schemes directly train on bounds from network relaxations to obtain models that are certifiably robust, but display sub-par standard performance. Recent work has shown that state-of-the-art trade-offs between certified robustness and standard performance can be obtained through a family of losses combining adversarial outputs and neural network bounds. Nevertheless, differently from empirical robustness, verifiability still comes at a significant cost in standard performance. In this work, we propose to leverage empirically-robust teachers to improve the performance of certifiably-robust models through knowledge distillation. Using a versatile feature-space distillation objective, we show that distillation from adversarially-trained teachers consistently improves on the state-of-the-art in certified training for ReLU networks across a series of robust computer vision benchmarks.
Paper Structure (46 sections, 6 theorems, 39 equations, 1 figure, 13 tables, 1 algorithm)

This paper contains 46 sections, 6 theorems, 39 equations, 1 figure, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Neural network verification
  4. Certified training
  5. Knowledge distillation
  6. Distillation for certified robustness
  7. Expressive losses: CC-IBP
  8. Distillation loss
  9. CC-Dist
  10. CC-Dist loss
  11. Teacher models
  12. Intuition
  13. Wider applicability
  14. Related work
  15. Experimental evaluation
  16. ...and 31 more sections

Key Result

Proposition 3.0

Let $\underaccent{\bar{}}{h}^{\mathcal{C}_{\epsilon}}_{\bm{\theta}_h}(\mathbf{x})$ and $\bar{h}^{\mathcal{C}_{\epsilon}}_{\bm{\theta}_h}(\mathbf{x})$ respectively denote IBP lower and upper bounds to the student features: The loss function is an upper bound to the worst-case distillation loss from equation eq:ideal-distillation-loss: $\bar{\mathcal{R}}_{f_{\bm{\theta}}}^{\mathcal{C}_{\epsilon}}

Figures (1)

  • Figure 1: Standard, empirical adversarial and certified accuracies (BaB and CROWN/IBP) under $\ell_\infty$ perturbations of networks trained using CC-Dist under varying distillation coefficient $\beta$. The legend is reported once for all subfigures in plot \ref{['fig:alpha-sensitivity-8-255']}. Metrics are reported on the CIFAR-10 test set. The $\beta$ value employed throughout the paper (see $\S$\ref{['sec:ccdist']}) is marked by a dashed vertical line.

Theorems & Definitions (10)

  • Proposition 3.0
  • Proposition 3.0
  • Lemma 3.0
  • Remark 2.1
  • Proposition 2.1
  • proof
  • Proposition 2.1
  • proof
  • Lemma 2.1
  • proof