Improved Scalable Lipschitz Bounds for Deep Neural Networks
Usman Syed, Bin Hu
TL;DR
This paper tackles the computation of global Lipschitz constants for deep neural networks, highlighting the limitations of SDP-based LipSDP in scalability and the conservatism of simple matrix-norm bounds. It develops a new family of SDP-free Lipschitz bounds by leveraging feasible points of LipSDP and parameterizing them with diagonal matrices, extending the ECLipsE-Fast framework. The main contributions include a constructive characterization of LipSDP-feasible points that subsume existing bounds, plus concrete bound families (ECLipsE-SN, ECLipsE-GC, ECLipsE-GCS, ECLipsE-Shift) with scalable computation and improved tightness. Empirical results on MNIST networks and randomly generated deep networks demonstrate consistently tighter bounds than ECLipsE-Fast while preserving (or improving) computational efficiency, underscoring the practical impact for scalable robustness verification of large neural nets.
Abstract
Computing tight Lipschitz bounds for deep neural networks is crucial for analyzing their robustness and stability, but existing approaches either produce relatively conservative estimates or rely on semidefinite programming (SDP) formulations (namely the LipSDP condition) that face scalability issues. Building upon ECLipsE-Fast, the state-of-the-art Lipschitz bound method that avoids SDP formulations, we derive a new family of improved scalable Lipschitz bounds that can be combined to outperform ECLipsE-Fast. Specifically, we leverage more general parameterizations of feasible points of LipSDP to derive various closed-form Lipschitz bounds, avoiding the use of SDP solvers. In addition, we show that our technique encompasses ECLipsE-Fast as a special case and leads to a much larger class of scalable Lipschitz bounds for deep neural networks. Our empirical study shows that our bounds improve ECLipsE-Fast, further advancing the scalability and precision of Lipschitz estimation for large neural networks.