Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Computable Lipschitz Bounds for Deep Neural Networks

Moreno Pintore, Bruno Després

TL;DR

One of the upper bounds of the Lipschitz constant of deep neural networks is optimal in the sense that it is exact for the first test with the simplest analytical form and it is better than other bounds for the other tests.

Abstract

Deriving sharp and computable upper bounds of the Lipschitz constant of deep neural networks is crucial to formally guarantee the robustness of neural-network based models. We analyse three existing upper bounds written for the $l^2$ norm. We highlight the importance of working with the $l^1$ and $l^\infty$ norms and we propose two novel bounds for both feed-forward fully-connected neural networks and convolutional neural networks. We treat the technical difficulties related to convolutional neural networks with two different methods, called explicit and implicit. Several numerical tests empirically confirm the theoretical results, help to quantify the relationship between the presented bounds and establish the better accuracy of the new bounds. Four numerical tests are studied: two where the output is derived from an analytical closed form are proposed; another one with random matrices; and the last one for convolutional neural networks trained on the MNIST dataset. We observe that one of our bound is optimal in the sense that it is exact for the first test with the simplest analytical form and it is better than other bounds for the other tests.

Computable Lipschitz Bounds for Deep Neural Networks

TL;DR

One of the upper bounds of the Lipschitz constant of deep neural networks is optimal in the sense that it is exact for the first test with the simplest analytical form and it is better than other bounds for the other tests.

Abstract

Deriving sharp and computable upper bounds of the Lipschitz constant of deep neural networks is crucial to formally guarantee the robustness of neural-network based models. We analyse three existing upper bounds written for the norm. We highlight the importance of working with the and norms and we propose two novel bounds for both feed-forward fully-connected neural networks and convolutional neural networks. We treat the technical difficulties related to convolutional neural networks with two different methods, called explicit and implicit. Several numerical tests empirically confirm the theoretical results, help to quantify the relationship between the presented bounds and establish the better accuracy of the new bounds. Four numerical tests are studied: two where the output is derived from an analytical closed form are proposed; another one with random matrices; and the last one for convolutional neural networks trained on the MNIST dataset. We observe that one of our bound is optimal in the sense that it is exact for the first test with the simplest analytical form and it is better than other bounds for the other tests.

Paper Structure

This paper contains 15 sections, 16 theorems, 99 equations, 7 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Fully-connected feed-forward neural networks
  3. Previous works
  4. New certified bounds
  5. Convolutional neural networks
  6. Explicit approach
  7. Explicit CNN bounds
  8. Implicit approach
  9. Implicit CNN bounds
  10. Numerical results
  11. Neural networks with random weights
  12. Approximation of $x^2$
  13. Approximation of $xy$
  14. Convolutional neural networks
  15. Conclusion

Key Result

Theorem 2.4

\newlabeldef:kcp0 One has the bound $K\leq K_1$ where

Figures (7)

  • Figure 1: Graphical representation of $g_1$, $g_2$ and $g_3$.
  • Figure 2: Graphical representation of the network representing the function $x - \sum_{r=0}^3 \frac{g_r(x)}{4^r}$. The blue dots and edges are associated with the construction of the function $g$. The red ones are used to store and sum $x$ and $-g_i/4^i$, $i=1,2,3$. The weight associated with the dashed lines is 0.
  • Figure 3: Lipschitz bounds for networks approximating the function $x^2$. The function $g$ is represented as in \ref{['eq:g_vers1']} (left) or as in \ref{['eq:g_vers2']} (right).
  • Figure 4: Rates of growth for networks approximating the function $x^2$. The function $g$ is represented as in \ref{['eq:g_vers1']} (left) or as in \ref{['eq:g_vers2']} (right).
  • Figure 5: Mesh used to construct the neural network approximating the function $xy$ on $[-1,1]^2$.
  • ...and 2 more figures

Theorems & Definitions (48)

  • Remark 2.1: Complexity
  • Definition 2.2: Worst bound
  • Definition 2.3
  • Theorem 2.4: Combettes-Pesquet bound
  • Remark 2.5
  • Remark 2.6
  • Proof 1
  • Proposition 2.7
  • Proof 2
  • Theorem 2.8: Virmaux-Scaman bound
  • ...and 38 more