FxTS-Net: Fixed-Time Stable Learning Framework for Neural ODEs

TL;DR

A new method for training Neural ODEs using fixed-time stability (FxTS) Lyapunov conditions based on the novel FxTS loss designed on Lyapunov functions, which aims to encourage convergence to accurate predictions in a user-defined fixed time.

Abstract

Neural Ordinary Differential Equations (Neural ODEs), as a novel category of modeling big data methods, cleverly link traditional neural networks and dynamical systems. However, it is challenging to ensure the dynamics system reaches a correctly predicted state within a user-defined fixed time. To address this problem, we propose a new method for training Neural ODEs using fixed-time stability (FxTS) Lyapunov conditions. Our framework, called FxTS-Net, is based on the novel FxTS loss (FxTS-Loss) designed on Lyapunov functions, which aims to encourage convergence to accurate predictions in a user-defined fixed time. We also provide an innovative approach for constructing Lyapunov functions to meet various tasks and network architecture requirements, achieved by leveraging supervised information during training. By developing a more precise time upper bound estimation for bounded non-vanishingly perturbed systems, we demonstrate that minimizing FxTS-Loss not only guarantees FxTS behavior of the dynamics but also input perturbation robustness. For optimising FxTS-Loss, we also propose a learning algorithm, in which the simulated perturbation sampling method can capture sample points in critical regions to approximate FxTS-Loss. Experimentally, we find that FxTS-Net provides better prediction performance and better robustness under input perturbation.

Figures (4)

• Figure 1: Comparing learned dynamics (plotting inference time vs prediction loss) on 1000 test examples from CIFAR-100. The orange line denotes standard Neural ODEs, the green line denotes FxTS-Net, and the corresponding light-colored areas denote the fluctuating regions across the samples. (a): Experiments on normal samples show FxTS-Net has a faster convergence rate and more stable evolutionary behavior compared to Neural ODEs. (b) Experiments on adversarial samples indicate that Neural ODEs without fixed-time stabilization result in significant distortions in even small input perturbations.
• Figure 2: The overall architecture of FxTS-Net: Using ResNet18 as the feature extractor, a data-controlled Neural ODE as the ODE-Block, a classifier as the output layer, and the corresponding model structure and computational data flow are plotted. Furthermore, on top of the ODE-Block, we plot a phase space, along with the level set of the Lyapunov function $V$, which in this example is minimized at $h^{*}$. In the phase space, the blue curve indicates the standard integration trajectory, the orange indicates the integration trajectory after perturbing the extracted features, and the red arrow indicates the descent direction of the optimization problem in Equation \ref{['eq:h']}.
• Figure 3: Plots of Scale Functions (setting $a_1,a_2,\delta=1$, and $\mu=4$). The red dotted line indicates error-free, i.e., multiplied by $1$ on $\delta$. The orange solid line indicates the scaling technique in garg2021advances, which multiplying $V$ on $\delta$. The remaining solid lines indicate ours.
• Figure 4: TSNE visualisation results on classification features of the Neural ODE and FxTS-Net using CIFAR10 with 400 random images per category. From left to right, FGSM attack perturbation, Gaussian noise perturbation and normal data.

Theorems & Definitions (11)

• Definition 2.1: Lyapunov Function
• Definition 2.2: FxTS
• Lemma 2.1: FxTS ConditionGARG2022110314
• Definition 3.1: Point-wise FxTS-Loss
• Definition 3.2: FxTS-Loss
• Theorem 3.1
• Proposition 3.1
• Theorem 3.2
• proof
• proof