Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ExLipBaB: Exact Lipschitz Constant Computation for Piecewise Linear Neural Networks

Tom A. Splittgerber

TL;DR

A generalization of the LipBaB algorithm is proposed to compute exact Lipschitz constants for arbitrary piecewise linear neural networks and $p-norms and may contain traditional activations like ReLU or LeakyReLU or the related MinMax and FullSort.

Abstract

It has been shown that a neural network's Lipschitz constant can be leveraged to derive robustness guarantees, to improve generalizability via regularization or even to construct invertible networks. Therefore, a number of methods varying in the tightness of their bounds and their computational cost have been developed to approximate the Lipschitz constant for different classes of networks. However, comparatively little research exists on methods for exact computation, which has been shown to be NP-hard. Nonetheless, there are applications where one might readily accept the computational cost of an exact method. These applications could include the benchmarking of new methods or the computation of robustness guarantees for small models on sensitive data. Unfortunately, existing exact algorithms restrict themselves to only ReLU-activated networks, which are known to come with severe downsides in the context of Lipschitz-constrained networks. We therefore propose a generalization of the LipBaB algorithm to compute exact Lipschitz constants for arbitrary piecewise linear neural networks and $p$-norms. With our method, networks may contain traditional activations like ReLU or LeakyReLU, activations like GroupSort or the related MinMax and FullSort, which have been of increasing interest in the context of Lipschitz constrained networks, or even other piecewise linear functions like MaxPool.

ExLipBaB: Exact Lipschitz Constant Computation for Piecewise Linear Neural Networks

TL;DR

A generalization of the LipBaB algorithm is proposed to compute exact Lipschitz constants for arbitrary piecewise linear neural networks and $p-norms and may contain traditional activations like ReLU or LeakyReLU or the related MinMax and FullSort.

Abstract

It has been shown that a neural network's Lipschitz constant can be leveraged to derive robustness guarantees, to improve generalizability via regularization or even to construct invertible networks. Therefore, a number of methods varying in the tightness of their bounds and their computational cost have been developed to approximate the Lipschitz constant for different classes of networks. However, comparatively little research exists on methods for exact computation, which has been shown to be NP-hard. Nonetheless, there are applications where one might readily accept the computational cost of an exact method. These applications could include the benchmarking of new methods or the computation of robustness guarantees for small models on sensitive data. Unfortunately, existing exact algorithms restrict themselves to only ReLU-activated networks, which are known to come with severe downsides in the context of Lipschitz-constrained networks. We therefore propose a generalization of the LipBaB algorithm to compute exact Lipschitz constants for arbitrary piecewise linear neural networks and -norms. With our method, networks may contain traditional activations like ReLU or LeakyReLU, activations like GroupSort or the related MinMax and FullSort, which have been of increasing interest in the context of Lipschitz constrained networks, or even other piecewise linear functions like MaxPool.
Paper Structure (27 sections, 6 theorems, 22 equations, 1 figure, 5 tables, 3 algorithms)

This paper contains 27 sections, 6 theorems, 22 equations, 1 figure, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Lipschitz constant and Jacobian
  4. Methodology
  5. Abstract interpretation
  6. Initialization
  7. Lipschitz Bounds
  8. Branching
  9. FFilter
  10. Conversion guarantee
  11. Application
  12. Absolute value function
  13. Real data applications
  14. Lower bound effectiveness
  15. Influence of input space size
  16. ...and 12 more sections

Key Result

Theorem 1

Let $\alpha:(\mathbb{R}^{d_1}, ||.||_p)\to (\mathbb{R}^{d_2}, ||.||_q)$ be a PWL function. Using the notation from definition def: methods-pwl, the following holds:

Figures (1)

  • Figure 1: Number of subproblems for ExLipBaB on the NN trained on wine data with and without initial lower bound.

Theorems & Definitions (16)

  • Definition 1
  • Definition 2
  • Remark 1
  • Theorem 1
  • Lemma 1: bhowmick_lipbab_2021
  • Lemma 2
  • proof
  • Definition S.1
  • Theorem S.1
  • proof
  • ...and 6 more