Dissecting Neural ODEs

TL;DR

This work provides a system-theoretic formulation of Neural ODEs, clarifying how depth-variance, augmentation, and training interact in continuous-depth models. It introduces two parameter-efficient depth-variant architectures (GalNODE and Stacked NODEs) and extends augmentation with input-layer and higher-order schemes, showing improved performance and efficiency. Moving beyond augmentation, the authors present data-control and adaptive-depth as powerful paradigms to learn complex maps and task-specific computation budgets, demonstrated through theoretical results and practical experiments. Together, these contributions deepen the understanding of continuous-depth models and expand their applicability to tasks requiring flexible depth and conditioning on data.

Abstract

Continuous deep learning architectures have recently re-emerged as Neural Ordinary Differential Equations (Neural ODEs). This infinite-depth approach theoretically bridges the gap between deep learning and dynamical systems, offering a novel perspective. However, deciphering the inner working of these models is still an open challenge, as most applications apply them as generic black-box modules. In this work we "open the box", further developing the continuous-depth formulation with the aim of clarifying the influence of several design choices on the underlying dynamics.

Figures (12)

• Figure 1: Galërkin Neural ODEs trained with integral losses accurately recover periodic signals. Blue curves correspond to different initial conditions and converge asymptotically to the reference desired trajectory.
• Figure 2: Galërkin and Stacked parameter-varying Neural ODE variants. Depth flows (Above) and evolution of the parameters (Below).
• Figure 3: Depth trajectories over vector field of the data--controlled neural ODEs (\ref{['eq:lin_sys']}) for $x=1,~x=-1$. The model learns a family of vector fields conditioned by the input $x$ to approximate $\varphi(x)$.
• Figure 4: Data--controlled CNFs can morph prior distributions into distinct posteriors to produce conditional samples. This task often requires crossing trajectories and is not possible with vanilla CNFs.
• Figure 5: Depth trajectories over vector field of the adaptive–-depth Neural ODEs. The reflection map can be learned by the proposed model. The key is to assign different integration times to the inputs, thus not requiring the intersection of trajectories.

Theorems & Definitions (14)

• Proposition 0: Generalized Adjoint Method
• theorem 1: Infinite--Dimensional Gradients
• corollary 1: Spectral Gradients
• corollary 2: Stacked Gradients
• Proposition 0
• Proposition 0: Generalized Adjoint Method
• Remark 1: Implementation of the generalized adjoint method
• theorem 1: Infinite--Dimensional Gradients
• corollary 2: Spectral Gradients
• Remark 2: Choose your parametrization