Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Designing Stable Neural Networks using Convex Analysis and ODEs

Ferdia Sherry, Elena Celledoni, Matthias J. Ehrhardt, Davide Murari, Brynjulf Owren, Carola-Bibiane Schönlieb

TL;DR

This work addresses the stability and robustness of deep networks by enforcing $1$-Lipschitz and averaged-operator properties through spectral-norm constraints, yielding ResNet-like blocks that mirror discretised gradient flows in convex potentials. It builds a theory-grounded architecture using explicit Runge–Kutta steps with circle contractivity to guarantee non-expansiveness, complemented by an adaptive training scheme for norm enforcement. Empirically, the approach delivers competitive performance on adversarial robustness (CIFAR-10), image denoising (BSDS500), and convergent Plug-and-Play denoising for inverse problems, while providing convergence guarantees for the learned denoisers. The work highlights practical feasibility for stable neural nets and suggests directions for applying these ideas to Wasserstein GANs and non-Euclidean norms, balancing stability with expressiveness in real-world tasks.

Abstract

Motivated by classical work on the numerical integration of ordinary differential equations we present a ResNet-styled neural network architecture that encodes non-expansive (1-Lipschitz) operators, as long as the spectral norms of the weights are appropriately constrained. This is to be contrasted with the ordinary ResNet architecture which, even if the spectral norms of the weights are constrained, has a Lipschitz constant that, in the worst case, grows exponentially with the depth of the network. Further analysis of the proposed architecture shows that the spectral norms of the weights can be further constrained to ensure that the network is an averaged operator, making it a natural candidate for a learned denoiser in Plug-and-Play algorithms. Using a novel adaptive way of enforcing the spectral norm constraints, we show that, even with these constraints, it is possible to train performant networks. The proposed architecture is applied to the problem of adversarially robust image classification, to image denoising, and finally to the inverse problem of deblurring.

Designing Stable Neural Networks using Convex Analysis and ODEs

TL;DR

This work addresses the stability and robustness of deep networks by enforcing -Lipschitz and averaged-operator properties through spectral-norm constraints, yielding ResNet-like blocks that mirror discretised gradient flows in convex potentials. It builds a theory-grounded architecture using explicit Runge–Kutta steps with circle contractivity to guarantee non-expansiveness, complemented by an adaptive training scheme for norm enforcement. Empirically, the approach delivers competitive performance on adversarial robustness (CIFAR-10), image denoising (BSDS500), and convergent Plug-and-Play denoising for inverse problems, while providing convergence guarantees for the learned denoisers. The work highlights practical feasibility for stable neural nets and suggests directions for applying these ideas to Wasserstein GANs and non-Euclidean norms, balancing stability with expressiveness in real-world tasks.

Abstract

Motivated by classical work on the numerical integration of ordinary differential equations we present a ResNet-styled neural network architecture that encodes non-expansive (1-Lipschitz) operators, as long as the spectral norms of the weights are appropriately constrained. This is to be contrasted with the ordinary ResNet architecture which, even if the spectral norms of the weights are constrained, has a Lipschitz constant that, in the worst case, grows exponentially with the depth of the network. Further analysis of the proposed architecture shows that the spectral norms of the weights can be further constrained to ensure that the network is an averaged operator, making it a natural candidate for a learned denoiser in Plug-and-Play algorithms. Using a novel adaptive way of enforcing the spectral norm constraints, we show that, even with these constraints, it is possible to train performant networks. The proposed architecture is applied to the problem of adversarially robust image classification, to image denoising, and finally to the inverse problem of deblurring.
Paper Structure (14 sections, 6 theorems, 44 equations, 6 figures, 2 tables, 3 algorithms)

This paper contains 14 sections, 6 theorems, 44 equations, 6 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related topics
  3. Our contributions
  4. Methods
  5. A more detailed look at the architecture for the forward Euler method
  6. Experiments
  7. Training methods
  8. Adversarial robustness
  9. Nonexpansive neural networks for denoising
  10. Plug-and-Play applications of the learned denoisers
  11. Conclusions and discussion
  12. Comparisons of computational costs
  13. Adversarial robustness examples
  14. Denoising examples

Key Result

Theorem 2.3

Suppose that $\Phi_h$ is an RK method satisfying the $r$-circle contractivity condition, and that $f$ satisfies the monotonicity condition Then, if $r\neq\infty$ and $h/r \leqslant 2\nu$, or if $r = \infty$ and $\nu \geqslant 0$,

Figures (6)

  • Figure 1: A comparison of the adversarial robustness of the proposed NonExpNet and a similar ResNet trained to classify CIFAR10 images. The baseline ResNet was trained both ordinarily (simply denoted "ResNet") and with adversarial training (denoted "ResNet-AT"). The dashed red line indicates the perturbation size that was used to generate the adversarial examples for adversarial training. The adversarial attack takes the form of an $\ell^2$-PGD attack with 100 iterations, with the robust accuracy curves computed on the test set.
  • Figure 2: Comparison of the denoising performance of the various considered approaches on the test set. For each image in the test set, 10 instances of noisy images were generated, so that each denoiser is tested on a total of 2000 noisy images. The learned approaches all perform similarly and outperform the variational approach that uses TV regularisation.
  • Figure 3: Comparison of some of the denoisers considered, on an image that is favourable to the learned approaches. Note that the learned approaches recover more fine details, whereas TV has a tendency to flatten them out. The numbers in the top right corner of each image are the PSNRs (in dB) relative to the ground truth $x$.
  • Figure 4: Comparison of some of the denoisers considered, on an image that is favourable to TV. The true image to be recovered is relatively well-approximated by a piecewise constant image. Note that the finer details in the image (for example the ridge between the top and bottom part of the beak) are still better recovered by the learned approaches. The numbers in the top right corner of each image are the PSNRs (in dB) relative to the ground truth $x$.
  • Figure 5: Repeated application of the unconstrained and constrained denoisers to a given input image gives drastically different results: for the unconstrained DnCNN, this sequence diverges, whereas for the averaged Euler denoiser, this sequence converges.
  • ...and 1 more figures

Theorems & Definitions (14)

  • Definition 2.1: Runge--Kutta method
  • Definition 2.2: Circle contractivity
  • Example 2.1
  • Remark 1
  • Theorem 2.3: Theorem 4.1 from dahlquist_generalized_1979
  • Theorem 2.4: Corollary 18.16 from bauschke_convex_2011
  • Lemma 2.5
  • proof
  • Definition 2.6: Definition 4.23 from bauschke_convex_2011
  • Lemma 2.7
  • ...and 4 more