Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Safety Verification of Deep Neural Networks

Xiaowei Huang, Marta Kwiatkowska, Sen Wang, Min Wu

TL;DR

This work tackles the safety verification of deep neural networks for image classification by formalizing a pointwise safety notion around a given input and solving the resulting verification problem via a novel SMT-based, layer-by-layer framework. It discretises perturbation regions, propagates region definitions across layers, and uses a ladder/variation formalism to guarantee exhaustive search for adversarial examples under user-defined manipulations. The authors implement a tool (DLV) using Z3 and demonstrate its ability to find adversarial examples and provide safety guarantees across datasets from MNIST to ImageNet, while comparing against established attack methods. The approach offers a principled path to certifiable safety for perception components in safety-critical systems, with potential for optimization and scalability through feature decomposition and abstraction refinement.

Abstract

Deep neural networks have achieved impressive experimental results in image classification, but can surprisingly be unstable with respect to adversarial perturbations, that is, minimal changes to the input image that cause the network to misclassify it. With potential applications including perception modules and end-to-end controllers for self-driving cars, this raises concerns about their safety. We develop a novel automated verification framework for feed-forward multi-layer neural networks based on Satisfiability Modulo Theory (SMT). We focus on safety of image classification decisions with respect to image manipulations, such as scratches or changes to camera angle or lighting conditions that would result in the same class being assigned by a human, and define safety for an individual decision in terms of invariance of the classification within a small neighbourhood of the original image. We enable exhaustive search of the region by employing discretisation, and propagate the analysis layer by layer. Our method works directly with the network code and, in contrast to existing methods, can guarantee that adversarial examples, if they exist, are found for the given region and family of manipulations. If found, adversarial examples can be shown to human testers and/or used to fine-tune the network. We implement the techniques using Z3 and evaluate them on state-of-the-art networks, including regularised and deep learning networks. We also compare against existing techniques to search for adversarial examples and estimate network robustness.

Safety Verification of Deep Neural Networks

TL;DR

This work tackles the safety verification of deep neural networks for image classification by formalizing a pointwise safety notion around a given input and solving the resulting verification problem via a novel SMT-based, layer-by-layer framework. It discretises perturbation regions, propagates region definitions across layers, and uses a ladder/variation formalism to guarantee exhaustive search for adversarial examples under user-defined manipulations. The authors implement a tool (DLV) using Z3 and demonstrate its ability to find adversarial examples and provide safety guarantees across datasets from MNIST to ImageNet, while comparing against established attack methods. The approach offers a principled path to certifiable safety for perception components in safety-critical systems, with potential for optimization and scalability through feature decomposition and abstraction refinement.

Abstract

Deep neural networks have achieved impressive experimental results in image classification, but can surprisingly be unstable with respect to adversarial perturbations, that is, minimal changes to the input image that cause the network to misclassify it. With potential applications including perception modules and end-to-end controllers for self-driving cars, this raises concerns about their safety. We develop a novel automated verification framework for feed-forward multi-layer neural networks based on Satisfiability Modulo Theory (SMT). We focus on safety of image classification decisions with respect to image manipulations, such as scratches or changes to camera angle or lighting conditions that would result in the same class being assigned by a human, and define safety for an individual decision in terms of invariance of the classification within a small neighbourhood of the original image. We enable exhaustive search of the region by employing discretisation, and propagate the analysis layer by layer. Our method works directly with the network code and, in contrast to existing methods, can guarantee that adversarial examples, if they exist, are found for the given region and family of manipulations. If found, adversarial examples can be shown to human testers and/or used to fine-tune the network. We implement the techniques using Z3 and evaluate them on state-of-the-art networks, including regularised and deep learning networks. We also compare against existing techniques to search for adversarial examples and estimate network robustness.

Paper Structure

This paper contains 32 sections, 3 theorems, 15 equations, 19 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background on Neural Networks
  3. Safety Analysis of Classification Decisions
  4. Safety and Robustness
  5. Manipulations
  6. Bounded Variation
  7. The Verification Framework
  8. Layer-by-Layer Analysis
  9. The Verification Method
  10. Feature Decomposition and Discovery
  11. Selection of Regions and Manipulations
  12. Determining the region $\eta_k$ according to $\eta_{k-1}$
  13. Determining the manipulation set $\Delta_k$ according to $\eta_k(\alpha_{x,k})$, $\eta_{k-1}(\alpha_{x,k-1})$, and $\Delta_{k-1}$
  14. Mapping Back to Input Layer
  15. Experimental Results
  16. ...and 17 more sections

Key Result

theorem thmcountertheorem

Given a neural network $N$, an input $x$, and a region $\eta_k$, we have that $N,\eta_k\models x$ implies $N,\eta_k,\Delta_k\models x$ for any set of manipulations $\Delta_k$.

Figures (19)

  • Figure 1: Automobile images (classified correctly) and their perturbed images (classified wrongly)
  • Figure 2: Example of a set $\{\delta_1,\delta_2,\delta_3,\delta_4\}$ of valid manipulations in a 2-dimensional space
  • Figure 3: Examples of ladders in region $\eta_k(\alpha_{x,k})$. Starting from $\alpha_{x,k}=\alpha_{x_0,k}$, the activations $\alpha_{x_1,k}... \alpha_{x_j,k}$ form a ladder such that each consecutive activation results from some valid manipulation $\delta_k$ applied to a previous activation, and the final activation $\alpha_{x_j,k}$ is outside the region $\eta_k(\alpha_{x,k})$.
  • Figure 4: Layer by layer analysis according to Definition \ref{['def:ek']}
  • Figure 5: Refinement framework
  • ...and 14 more figures

Theorems & Definitions (12)

  • definition thmcounterdefinition
  • definition thmcounterdefinition
  • definition thmcounterdefinition
  • definition thmcounterdefinition
  • definition thmcounterdefinition
  • theorem thmcountertheorem
  • theorem thmcountertheorem
  • definition thmcounterdefinition
  • definition thmcounterdefinition
  • theorem thmcountertheorem
  • ...and 2 more