Neural Fractional Differential Equations

TL;DR

This work extends Neural ODEs by introducing Neural Fractional Differential Equations that replace the local time derivative with a Caputo fractional derivative $\, _{0}^{C}D^{\alpha}_{t}$ of order $\alpha \in (0,1)$, where $\alpha$ is learned by a dedicated neural network. The method combines two neural networks, $\mathbf{f}_{\boldsymbol{\theta}}$ for the dynamics and $\alpha_{\boldsymbol{\phi}}$ for the derivative order, with a Predictor-Corrector solver to compute solutions and enable backpropagation. Across synthetic and real-world datasets, Neural FDEs show improved accuracy and notably faster convergence in memory-rich tasks compared to Neural ODEs, albeit with higher computational cost due to nonlocal history dependence. The study highlights the potential of nonlocal dynamics for time-series modelling and outlines avenues to improve efficiency and stability, such as graded meshes and alternative loss formulations for $\alpha$.

Abstract

Fractional Differential Equations (FDEs) are essential tools for modelling complex systems in science and engineering. They extend the traditional concepts of differentiation and integration to non-integer orders, enabling a more precise representation of processes characterised by non-local and memory-dependent behaviours. This property is useful in systems where variables do not respond to changes instantaneously, but instead exhibit a strong memory of past interactions. Having this in mind, and drawing inspiration from Neural Ordinary Differential Equations (Neural ODEs), we propose the Neural FDE, a novel deep neural network architecture that adjusts a FDE to the dynamics of data. This work provides a comprehensive overview of the numerical method employed in Neural FDEs and the Neural FDE architecture. The numerical outcomes suggest that, despite being more computationally demanding, the Neural FDE may outperform the Neural ODE in modelling systems with memory or dependencies on past states, and it can effectively be applied to learn more intricate dynamical systems.

Figures (10)

• Figure 1: Fit of an ODE to data $\{\boldsymbol{x}_0, \boldsymbol{x}_1, \boldsymbol{x}_2, \boldsymbol{x}_3, \boldsymbol{x}_4 \}$ obtained experimentally or provided by a dynamical system. The blue symbols represent the data points, while the orange line represents the fit obtained from the initial value problem shown on the right. Each vector $\boldsymbol{x}_i$ corresponds to a specific instant $t_i$. The initial value problem allows us to determine the behaviour of the dynamical system at any instant within the interval $[t_0, T]$.
• Figure 2: Example of a typical mesh used in the numerical solution of Eq. \ref{['eq:NeuralODE']} for each interval $[t_i,t_{i+1}]$, $i=0,\dots, N-1$, where $t_i$ is the time of observation $\boldsymbol{x}_i$.
• Figure 3: Schematic of a Neural ODE iteration. Note that along the sequence of figures (left to right) the NN $\boldsymbol{f}_{\boldsymbol{\theta}}(t, \boldsymbol{h}(t))$ doesn't change.
• Figure 4: Memory in FDEs (left) and ODEs (right). Schematic of the computation of the derivative (fractional or classical) at instant $t$. The case of short memory (center) is presented for illustrative purposes, aiming to facilitate a clear understanding of this phenomenon.
• Figure 5: Schematic of a Neural ODE iteration. Note that along the sequence of figures (left to right) the NNs $\boldsymbol{f}_{\boldsymbol{\theta}}(t, \boldsymbol{h}(t))$ and $\alpha_{\boldsymbol{\phi}}$ don't change. Note also that in the next iteration, the output $\alpha_{\boldsymbol{out}}$ will be used as the input for $\alpha_{\boldsymbol{\phi}}$ throughout the entire new iteration.

Theorems & Definitions (6)

• Theorem 1
• Lemma 2
• Definition 1
• Lemma 3
• Definition 2
• Definition 3