LipKernel: Lipschitz-Bounded Convolutional Neural Networks via Dissipative Layers

TL;DR

A novel layer-wise parameterization for convolutional neural networks (CNNs) that includes built-in robustness guarantees by enforcing a prescribed Lipschitz bound is proposed, aimed at improving robustness of learning-based real-time perception or control in robotics, autonomous vehicles, or automation systems.

Abstract

We propose a novel layer-wise parameterization for convolutional neural networks (CNNs) that includes built-in robustness guarantees by enforcing a prescribed Lipschitz bound. Each layer in our parameterization is designed to satisfy a linear matrix inequality (LMI), which in turn implies dissipativity with respect to a specific supply rate. Collectively, these layer-wise LMIs ensure Lipschitz boundedness for the input-output mapping of the neural network, yielding a more expressive parameterization than through spectral bounds or orthogonal layers. Our new method LipKernel directly parameterizes dissipative convolution kernels using a 2-D Roesser-type state space model. This means that the convolutional layers are given in standard form after training and can be evaluated without computational overhead. In numerical experiments, we show that the run-time using our method is orders of magnitude faster than state-of-the-art Lipschitz-bounded networks that parameterize convolutions in the Fourier domain, making our approach particularly attractive for improving robustness of learning-based real-time perception or control in robotics, autonomous vehicles, or automation systems. We focus on CNNs, and in contrast to previous works, our approach accommodates a wide variety of layers typically used in CNNs, including 1-D and 2-D convolutional layers, maximum and average pooling layers, as well as strided and dilated convolutions and zero padding. However, our approach naturally extends beyond CNNs as we can incorporate any layer that is incrementally dissipative.

Figures (5)

• Figure 1: For $\mathcal{F}_2\circ\sigma\circ\mathcal{F}_1$ with $c_0=c_1=c_2=2$, we compare over-approximations for reachability sets shown in blue, we obtain ellipsoidal sets using incrementally dissipative layers (top) and circles using Lipschitz bounds (bottom).
• Figure 2: Fit of a cosine function using NN from LMI-based parameterization with dissipative layers and an NN with 1-Lipschitz layers with weights which are constrained to have spectral norm $1$.
• Figure 3: Differences between convolutional layers using LipKernel (ours) and Sandwich layers wang2023direct in its parameterization complexity and its standard evaluation. The light blue boxes represent images and the green boxes the kernel.
• Figure 4: Inference times for LipKernel, Sandwich, and Orthogon layers with different number of channels $c=c_{\mathrm{in}}=c_{\mathrm{out}}$, input image sizes $N=N_1=N_2$ and kernel sizes $k=k_1=k_2$. For all layers, we have stride equal to $1$ and average the run-time over 10 different initializations.
• Figure 5: Robustness accuracy trade-off for 2C2F (left) 2CP2F (right) for NNs averaged over three initializations.

Theorems & Definitions (31)

• Remark 1
• Remark 2
• Definition 1: Incremental dissipativity byrnes1994losslessness
• Lemma 1: Realization of 2-D convolutions pauli2024state
• Remark 3
• Lemma 2: Slope-restriction fazlyab2019efficientpauli2021training
• Theorem 1
• proof
• Lemma 3: LMI for $\sigma\circ\mathcal{C}$ pauli2024lipschitz
• proof