A provable control of sensitivity of neural networks through a direct parameterization of the overall bi-Lipschitzness

TL;DR

This work tackles the challenge of understanding neural networks by enforcing a controllable bi-Lipschitz inductive bias through a novel BLNN framework grounded in convex neural networks and Legendre-Fenchel duality. The core idea yields a direct, two-parameter control of the overall Lipschitz and inverse Lipschitz constants, with rigorous guarantees on expressive power and a universal approximation property before differentiation. The authors demonstrate that a BLNN can robustly bound sensitivity, enable stable backward passes, and be efficiently extended to partially bi-Lipschitz variants for scalability. They validate the approach with experiments on uncertainty estimation and monotone problems, showing improved bound tightness, better out-of-distribution detection, and competitive performance in monotone settings. Overall, the BLNN framework provides a principled, analyzable path to harness bi-Lipschitzness in neural networks for robustness and reliability in practical tasks.

Abstract

While neural networks can enjoy an outstanding flexibility and exhibit unprecedented performance, the mechanism behind their behavior is still not well-understood. To tackle this fundamental challenge, researchers have tried to restrict and manipulate some of their properties in order to gain new insights and better control on them. Especially, throughout the past few years, the concept of \emph{bi-Lipschitzness} has been proved as a beneficial inductive bias in many areas. However, due to its complexity, the design and control of bi-Lipschitz architectures are falling behind, and a model that is precisely designed for bi-Lipschitzness realizing a direct and simple control of the constants along with solid theoretical analysis is lacking. In this work, we investigate and propose a novel framework for bi-Lipschitzness that can achieve such a clear and tight control based on convex neural networks and the Legendre-Fenchel duality. Its desirable properties are illustrated with concrete experiments. We also apply this framework to uncertainty estimation and monotone problem settings to illustrate its broad range of applications.

Figures (27)

• Figure 1: Results of fitting $y=50x$ with a Lipschitz model (SN (left) or our model (right)), where the Lipschitz constant is constrained by an upper bound $L$. $L=50$ (red line) is where an $L$-Lipschitz model with perfect tightness and expressive power should achieve a 0 loss for the first time. SN achieves this only from around $L=100$ while ours at $L=50$. See Appendix \ref{['ap:exp_sum']} for further details.
• Figure 2: Comparison of the time (left) and space (right) complexity for a single iteration between a traditional feedforward network and various BLNN variants.
• Figure 3: Results of fitting $f(x)=x\ (x<0),\ x+1\ (x\ge 0)$ with SLL (left), Sandwich (middle) and our method (right) with a specified Lipschitzness of 50. See Figure \ref{['fig:theory_toy']} for further details.
• Figure 4: Results of fitting the linear function $y=x$ with (from left) AOL, Sandwich, BiLipNet and our method with a specified Lipschitzness of 1000. See Figures \ref{['fig:uniform_toy1']} and \ref{['fig:uniform_toy2']} for further results.
• Figure 5: Uncertainty estimation with the two moons data set with several models. Blue indicates high uncertainty, and yellow low uncertainty. (d)-(f) are with DUQ+BLNN, where $(\alpha,\beta)$ are clarified.

Theorems & Definitions (63)

• Definition 2.1: bi-Lipschitzness
• Definition 3.1
• Definition 3.2
• Definition 3.3
• Theorem 3.4
• Theorem 3.5
• Theorem 3.6
• Theorem 3.7
• Definition A.1: Lipschitzness
• Definition A.2: inverse Lipschitzness