Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximation theory for 1-Lipschitz ResNets

Davide Murari, Takashi Furuya, Carola-Bibiane Schönlieb

TL;DR

We address the problem of universal approximation by $1$-Lipschitz ResNets built from explicit Euler steps for negative gradient flows. The paper proves two main results: first, density of networks with unbounded width and depth in the class of scalar $1$-Lipschitz functions on compact domains; second, density with a fixed hidden width achieved by inserting norm-constrained linear maps between residual blocks. A constructive, alternative view shows that all scalar $1$-Lipschitz piecewise affine functions lie in the network class, strengthening the universality claim. The framework enables implementable architectures that can be trained with standard optimizers while preserving the $1$-Lipschitz property, supporting robust learning, inverse problems, and generative modelling.

Abstract

1-Lipschitz neural networks are fundamental for generative modelling, inverse problems, and robust classifiers. In this paper, we focus on 1-Lipschitz residual networks (ResNets) based on explicit Euler steps of negative gradient flows and study their approximation capabilities. Leveraging the Restricted Stone-Weierstrass Theorem, we first show that these 1-Lipschitz ResNets are dense in the set of scalar 1-Lipschitz functions on any compact domain when width and depth are allowed to grow. We also show that these networks can exactly represent scalar piecewise affine 1-Lipschitz functions. We then prove a stronger statement: by inserting norm-constrained linear maps between the residual blocks, the same density holds when the hidden width is fixed. Because every layer obeys simple norm constraints, the resulting models can be trained with off-the-shelf optimisers. This paper provides the first universal approximation guarantees for 1-Lipschitz ResNets, laying a rigorous foundation for their practical use.

Approximation theory for 1-Lipschitz ResNets

TL;DR

We address the problem of universal approximation by -Lipschitz ResNets built from explicit Euler steps for negative gradient flows. The paper proves two main results: first, density of networks with unbounded width and depth in the class of scalar -Lipschitz functions on compact domains; second, density with a fixed hidden width achieved by inserting norm-constrained linear maps between residual blocks. A constructive, alternative view shows that all scalar -Lipschitz piecewise affine functions lie in the network class, strengthening the universality claim. The framework enables implementable architectures that can be trained with standard optimizers while preserving the -Lipschitz property, supporting robust learning, inverse problems, and generative modelling.

Abstract

1-Lipschitz neural networks are fundamental for generative modelling, inverse problems, and robust classifiers. In this paper, we focus on 1-Lipschitz residual networks (ResNets) based on explicit Euler steps of negative gradient flows and study their approximation capabilities. Leveraging the Restricted Stone-Weierstrass Theorem, we first show that these 1-Lipschitz ResNets are dense in the set of scalar 1-Lipschitz functions on any compact domain when width and depth are allowed to grow. We also show that these networks can exactly represent scalar piecewise affine 1-Lipschitz functions. We then prove a stronger statement: by inserting norm-constrained linear maps between the residual blocks, the same density holds when the hidden width is fixed. Because every layer obeys simple norm constraints, the resulting models can be trained with off-the-shelf optimisers. This paper provides the first universal approximation guarantees for 1-Lipschitz ResNets, laying a rigorous foundation for their practical use.
Paper Structure (42 sections, 29 theorems, 93 equations, 3 tables)

This paper contains 42 sections, 29 theorems, 93 equations, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. $1$-Lipschitz neural networks.
  4. Approximation theory for Lipschitz-constrained networks.
  5. Main contributions
  6. Outline of the paper
  7. Preliminaries
  8. Universal approximation property
  9. $1$-Lipschitz residual layers
  10. Density with unbounded width and depth
  11. Proof of Theorem \ref{['thm:gradFreeInner']} based on Restricted Stone-Weierstrass
  12. Proof of Theorem \ref{['thm:gradFreeInner']} based on piecewise affine functions
  13. Density with fixed width and unbounded depth
  14. Proof of Theorem \ref{['thm:gradFixedInner']}
  15. Discussion and future work
  16. ...and 27 more sections

Key Result

Theorem 2.1

Let $\mathcal{X}\subset\mathbb{R}^d$ be compact and have at least two points. Let $\mathcal{A}\subset\mathcal{C}_1(\mathcal{X},\mathbb{R})$ be a lattice separating the points of $\mathcal{X}$. Then $\mathcal{A}$ satisfies the universal approximation property for $\mathcal{C}_1(\mathcal{X},\mathbb{R}

Theorems & Definitions (49)

  • Definition 2.1
  • Definition 2.2: Lattice
  • Definition 2.3: Subset separating points
  • Theorem 2.1: Restricted Stone-Weierstrass
  • Proposition 2.1: Theorem 2.3 and Lemma 2.5 in sherry2024designing
  • Lemma 2.1
  • proof
  • Theorem 3.1
  • Lemma 3.1
  • Lemma 3.2
  • ...and 39 more