Chordal Sparsity for Lipschitz Constant Estimation of Deep Neural Networks

TL;DR

This work addresses the computational bottleneck of Lipschitz constant estimation for deep neural networks by leveraging chordal sparsity to decompose the large semidefinite constraint in LipSDP into many smaller constraints, yielding Chordal-LipSDP. The authors characterize the sparsity pattern via a tunable parameter $\tau$, prove that the chordal decomposition is equivalent to the original LipSDP (preserving the bound $L \le (\gamma_\ell^\star)^{1/2}$), and demonstrate substantial scalability improvements on deep networks with controllable bound tightening. Empirical results show faster runtimes for deeper networks and that increasing $\tau$ yields significantly tighter Lipschitz estimates without large performance penalties. The method preserves accuracy while enabling scalable robustness certificates, and the authors provide open-source implementation for broader adoption.

Abstract

Lipschitz constants of neural networks allow for guarantees of robustness in image classification, safety in controller design, and generalizability beyond the training data. As calculating Lipschitz constants is NP-hard, techniques for estimating Lipschitz constants must navigate the trade-off between scalability and accuracy. In this work, we significantly push the scalability frontier of a semidefinite programming technique known as LipSDP while achieving zero accuracy loss. We first show that LipSDP has chordal sparsity, which allows us to derive a chordally sparse formulation that we call Chordal-LipSDP. The key benefit is that the main computational bottleneck of LipSDP, a large semidefinite constraint, is now decomposed into an equivalent collection of smaller ones: allowing Chordal-LipSDP to outperform LipSDP particularly as the network depth grows. Moreover, our formulation uses a tunable sparsity parameter that enables one to gain tighter estimates without incurring a significant computational cost. We illustrate the scalability of our approach through extensive numerical experiments.

Figures (5)

• Figure 1: The sparsity of $Z(\gamma)$ for $\tau = 0, 2, 4$ with dimensions $(3,3,3,3,3)$. For each increment of $\tau$, each block grows by one unit on the bottom and right, and corresponds to a maximal clique of ${\mathcal{G}}({\mathcal{V}}, {\mathcal{E}})$. As $\tau$ increases the number of blocks (maximal cliques) will decrease as the lower-right blocks become overshadowed. At $\tau = 0$ we have what fazlyab2019efficient refers to as "LipSDP-neuron"; at $\tau = N_f - 1$ we have the completely dense "LipSDP-network".
• Figure 2: The runtimes (seconds) of Chordal-LipSDP, LipSDP, and CP-Lip on a subset of the networks. The times for Naive-Lip are omitted because it finishes in $< 0.1$ seconds on all instances. We ran Chordal-LipSDP and LipSDP for $\tau = 0, \ldots, 6$. Because CP-Lip is independent of $\tau$, it is a constant line. Moreover, due to the scaling exponentially with respect to the number of layers, we only ran CP-Lip for networks of depth $\leq 25$.
• Figure 3: The runtimes (seconds) of LipSDP and Chordal-LipSDP as the depth varies. Each plot shows networks that share the same width, but whose depths are varied on the x-axis. Each curve shows the runtimes for a different value of $\tau = 0, \ldots, 6$, with higher curves corresponding to higher runtimes --- and in this case also higher values of $\tau$. We shade the region between the $\tau = 0$ and $\tau = 6$ curves for each method.
• Figure 4: The runtimes (seconds) of LipSDP and Chordal-LipSDP as the width varies. Similar to Figure \ref{['fig:fixed-widths']}, but the x-axis now shows varying widths.
• Figure 5: The Lipschitz constant estimate given by LipSDP (the same as Chordal-LipSDP) on some networks, with $\tau$ on the x-axis. On the left we also plot the estimates given by CP-Lip (green) and Naive-Lip (purple).

Theorems & Definitions (16)

• Lemma 1: Theorem 2.10 zheng2019chordal
• Theorem 1
• Theorem 2
• Theorem 3
• Lemma 2
• proof
• Lemma 3
• proof
• Lemma 4
• proof