ECLipsE-Gen-Local: Efficient Compositional Local Lipschitz Estimates for Deep Neural Networks
Yuezhu Xu, S. Sivaranjani
TL;DR
This work tackles the NP-hard problem of certifying local Lipschitz constants for deep neural networks by extending the LipSDP framework to heterogeneous activation slope bounds and arbitrary input-output indices, then decomposing the global certificate into small, layer-wise subproblems. By incorporating local input information, the proposed ECLipsE-Gen-Local framework yields substantially tighter local Lipschitz bounds than global methods, with linear scaling in network depth and a closed-form variant for near-instant computation. The authors provide theoretical guarantees on feasibility and tightness, and demonstrate through extensive experiments that local certificates closely track the Jacobian norm as the input region shrinks and that robustness-trained networks exhibit smaller Lipschitz bounds and improved adversarial robustness. This approach enables scalable, verifiable Lipschitz certificates suitable for safety-critical applications and reachability analyses, with potential integration into robust training pipelines.
Abstract
The Lipschitz constant is a key measure for certifying the robustness of neural networks to input perturbations. However, computing the exact constant is NP-hard, and standard approaches to estimate the Lipschitz constant involve solving a large matrix semidefinite program (SDP) that scales poorly with network size. Further, there is a potential to efficiently leverage local information on the input region to provide tighter Lipschitz estimates. We address this problem here by proposing a compositional framework that yields tight yet scalable Lipschitz estimates for deep feedforward neural networks. Specifically, we begin by developing a generalized SDP framework that is highly flexible, accommodating heterogeneous activation function slope, and allowing Lipschitz estimates with respect to arbitrary input-output pairs and arbitrary choices of sub-networks of consecutive layers. We then decompose this generalized SDP into a sequence of small sub-problems, with computational complexity that scales linearly with respect to the network depth. We also develop a variant that achieves near-instantaneous computation through closed-form solutions to each sub-problem. All our algorithms are accompanied by theoretical guarantees on feasibility and validity. Next, we develop a series of algorithms, termed as ECLipsE-Gen-Local, that effectively incorporate local information on the input. Our experiments demonstrate that our algorithms achieve substantial speedups over a multitude of benchmarks while producing significantly tighter Lipschitz bounds than global approaches. Moreover, we show that our algorithms provide strict upper bounds for the Lipschitz constant with values approaching the exact Jacobian from autodiff when the input region is small enough. Finally, we demonstrate the practical utility of our approach by showing that our Lipschitz estimates closely align with network robustness.