Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Continuous-Time Piecewise-Linear Recurrent Neural Networks

Alena Brändle, Lukas Eisenmann, Florian Götz, Daniel Durstewitz

TL;DR

A novel algorithm is presented for training and simulating continuous-time PLRNNs (cPLRNNs), bypassing numerical integration by efficiently exploiting their PL structure, and it is demonstrated how important topological objects like equilibria or limit cycles can be determined semi-analytically in trained models.

Abstract

In dynamical systems reconstruction (DSR) we aim to recover the dynamical system (DS) underlying observed time series. Specifically, we aim to learn a generative surrogate model which approximates the underlying, data-generating DS, and recreates its long-term properties (`climate statistics'). In scientific and medical areas, in particular, these models need to be mechanistically tractable -- through their mathematical analysis we would like to obtain insight into the recovered system's workings. Piecewise-linear (PL), ReLU-based RNNs (PLRNNs) have a strong track-record in this regard, representing SOTA DSR models while allowing mathematical insight by virtue of their PL design. However, all current PLRNN variants are discrete-time maps. This is in disaccord with the assumed continuous-time nature of most physical and biological processes, and makes it hard to accommodate data arriving at irregular temporal intervals. Neural ODEs are one solution, but they do not reach the DSR performance of PLRNNs and often lack their tractability. Here we develop theory for continuous-time PLRNNs (cPLRNNs): We present a novel algorithm for training and simulating such models, bypassing numerical integration by efficiently exploiting their PL structure. We further demonstrate how important topological objects like equilibria or limit cycles can be determined semi-analytically in trained models. We compare cPLRNNs to both their discrete-time cousins as well as Neural ODEs on DSR benchmarks, including systems with discontinuities which come with hard thresholds.

Continuous-Time Piecewise-Linear Recurrent Neural Networks

TL;DR

A novel algorithm is presented for training and simulating continuous-time PLRNNs (cPLRNNs), bypassing numerical integration by efficiently exploiting their PL structure, and it is demonstrated how important topological objects like equilibria or limit cycles can be determined semi-analytically in trained models.

Abstract

In dynamical systems reconstruction (DSR) we aim to recover the dynamical system (DS) underlying observed time series. Specifically, we aim to learn a generative surrogate model which approximates the underlying, data-generating DS, and recreates its long-term properties (`climate statistics'). In scientific and medical areas, in particular, these models need to be mechanistically tractable -- through their mathematical analysis we would like to obtain insight into the recovered system's workings. Piecewise-linear (PL), ReLU-based RNNs (PLRNNs) have a strong track-record in this regard, representing SOTA DSR models while allowing mathematical insight by virtue of their PL design. However, all current PLRNN variants are discrete-time maps. This is in disaccord with the assumed continuous-time nature of most physical and biological processes, and makes it hard to accommodate data arriving at irregular temporal intervals. Neural ODEs are one solution, but they do not reach the DSR performance of PLRNNs and often lack their tractability. Here we develop theory for continuous-time PLRNNs (cPLRNNs): We present a novel algorithm for training and simulating such models, bypassing numerical integration by efficiently exploiting their PL structure. We further demonstrate how important topological objects like equilibria or limit cycles can be determined semi-analytically in trained models. We compare cPLRNNs to both their discrete-time cousins as well as Neural ODEs on DSR benchmarks, including systems with discontinuities which come with hard thresholds.
Paper Structure (56 sections, 50 equations, 4 figures, 6 tables, 2 algorithms)

This paper contains 56 sections, 50 equations, 4 figures, 6 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dynamical systems reconstruction (DSR)
  4. Continuous-time RNNs
  5. Piecewise linear systems
  6. Model Formulation and Theoretical Analysis
  7. Continuous PLRNN and Solution Method
  8. Data assumptions
  9. Model formulation
  10. Solving the cPLRNN equations
  11. Interval root finding
  12. Differentiating through root times
  13. Computing the states
  14. Sparse teacher forcing (STF)
  15. State Space Analysis of Trained Models
  16. ...and 41 more sections

Figures (4)

  • Figure 1: Example reconstructions for all three methods compared: Ground truth trajectories and fixed points in black for Lorenz-63 system, and model-generated trajectories ($M=20, \ P=10$ for all models) and corresponding fixed points found with SCYFI in red.
  • Figure 2: A) Top: LIF model (black) and trajectory generated by cPLRNN (red) with $M = 25$ and $P = 2$. Bottom: Limit cycle and fixed point found in cPLRNN trained on time series from LIF model. A) Top: Membrane potential recordings (black) and trajectory generated by cPLRNN (red) with $M = 25$ and $P = 6$. Bottom: Limit cycle and fixed point found in cPLRNN trained on empirical data.
  • Figure 3: Illustration of several roots in a non-bracketing interval.
  • Figure 4: Illustration of problem setting: for obtaining a global trajectory solution at arbitrary time points $t_i$ we need to find the switching times $t_{s,k}$ between linear subregions.