Stable neural networks and connections to continuous dynamical systems

TL;DR

This work argues that neural networks are prone to instabilities and adversarial perturbations, and it advocates a unified dynamical-systems framework to design stable architectures. It surveys three core approaches—non-expansive flows, Hamiltonian (symplectic) networks, and stability-based equilibria—and develops practical, Lipschitz-controlled layers (notably gradient-flow blocks and InvNet) with accompanying code. The paper demonstrates two worked examples: an ill-conditioned inverse problem and adversarially robust image classification, showing that stability guarantees can improve robustness without sacrificing performance on clean data. Overall, it provides a principled pathway from continuous dynamics to stable, trainable networks with tangible benefits for inverse problems and robustness in vision tasks.

Abstract

The existence of instabilities, for example in the form of adversarial examples, has given rise to a highly active area of research concerning itself with understanding and enhancing the stability of neural networks. We focus on a popular branch within this area which draws on connections to continuous dynamical systems and optimal control, giving a bird's eye view of this area. We identify and describe the fundamental concepts that underlie much of the existing work in this area. Following this, we go into more detail on a specific approach to designing stable neural networks, developing the theoretical background and giving a description of how these networks can be implemented. We provide code that implements the approach that can be adapted and extended by the reader. The code further includes a notebook with a fleshed-out toy example on adversarial robustness of image classification that can be run without heavy requirements on the reader's computer. We finish by discussing this toy example so that the reader can interactively follow along on their computer. This work will be included as a chapter of a book on scientific machine learning, which is currently under revision and aimed at students.

Figures (14)

• Figure 1: An illustration of how the presence of noise and ill-conditioning of the matrix $A$ combine to complicate the inversion process.
• Figure 2: MLP network with 12 layers trained to distinguish red from blue points. On the left, the learned decision boundary cannot accurately separate the points, resulting in a test accuracy of 51%. On the right, the norms of the Jacobians through the training iterations for a fixed data point, showing a severe attenuation of information as we progress through the network, an issue known as the vanishing gradient problem.
• Figure 3: A demonstration of the behaviours discussed in Exercise \ref{['exercise:ho_euler_symplectic']} for the harmonic oscillator, with an initial condition $x_0 = 1, p_0=0$.
• Figure 4: A comparison of a 12-layer HNN, a 12-layer ResNet and a 2-layer MLP on the "Swiss roll" dataset, as previously considered for a 12-layer MLP in Figure \ref{['fig:ex:training']}. Both the HNN and ResNet attain a test accuracy of 100%, while the 2-layer MLP has a test accuracy of 79.23%. Note that the Jacobian norms behave much less extremely than they did for the 12-layer MLP in Figure \ref{['fig:ex:training']}, resulting in networks that train better.
• Figure 5: Phase portrait of three linear systems. From left to right, the origin is asymptotically stable, unstable, and stable but not asymptotically.

Theorems & Definitions (6)

• Example 1: Inverse Problems
• Example 2: Classification
• Example 3: Stable Training
• Definition 1: One-sided Lipschitz inequality
• Definition 2
• Definition 3: Lyapunov function