Almost-Linear RNNs Yield Highly Interpretable Symbolic Codes in Dynamical Systems Reconstruction

TL;DR

Almost-Linear Recurrent Recurrent Neural Networks (AL-RNNs) are introduced which automatically and robustly produce most parsimonious PWL representations of DS from time series data, using as few PWL nonlinearities as possible.

Abstract

Dynamical systems (DS) theory is fundamental for many areas of science and engineering. It can provide deep insights into the behavior of systems evolving in time, as typically described by differential or recursive equations. A common approach to facilitate mathematical tractability and interpretability of DS models involves decomposing nonlinear DS into multiple linear DS separated by switching manifolds, i.e. piecewise linear (PWL) systems. PWL models are popular in engineering and a frequent choice in mathematics for analyzing the topological properties of DS. However, hand-crafting such models is tedious and only possible for very low-dimensional scenarios, while inferring them from data usually gives rise to unnecessarily complex representations with very many linear subregions. Here we introduce Almost-Linear Recurrent Neural Networks (AL-RNNs) which automatically and robustly produce most parsimonious PWL representations of DS from time series data, using as few PWL nonlinearities as possible. AL-RNNs can be efficiently trained with any SOTA algorithm for dynamical systems reconstruction (DSR), and naturally give rise to a symbolic encoding of the underlying DS that provably preserves important topological properties. We show that for the Lorenz and Rössler systems, AL-RNNs discover, in a purely data-driven way, the known topologically minimal PWL representations of the corresponding chaotic attractors. We further illustrate on two challenging empirical datasets that interpretable symbolic encodings of the dynamics can be achieved, tremendously facilitating mathematical and computational analysis of the underlying systems.

Figures (25)

• Figure 1: Illustration of the AL-RNN architecture.
• Figure 2: Illustration of symbolic approach (3 panels on the left) and geometrical graphs (right).
• Figure 3: Quantification of DSR quality in terms of attractor geometry disagreement ($D_{\text{stsp}}$, top row) and disagreement in temporal structure ($D_{\text{H}}$, bottom row) as a function of the number of ReLUs ($P$) in the AL-RNN ( Rössler: $M=20$, Lorenz-63: $M=20$, ECG: $M=100$, fMRI: $M=50$). The little humps at $P=3$ for the Lorenz-63 indicate that performance may sometimes first degrade again when passing the number of minimally necessary PWL units (see also Fig. \ref{['fig:regularization']}). Error bars = SEM.
• Figure 4: Left: Number of linear subregions traversed by trained AL-RNNs as a function of the number $P$ of ReLUs. Theoretical limit ($2^P$) in red. Right: Cumulative number of data (trajectory) points covered by linear subregions in trained AL-RNNs (Rössler: $M=20, P=10$, Lorenz-63: $M=20, P=10$, ECG: $M=100, P=10$), illustrating that trajectories on an attractor live in a relatively small subset of subregions.
• Figure 5: a: Color-coded linear subregions of minimal AL-RNNs representing the Rössler (top) and Lorenz-63 (bottom) chaotic attractor. b: Illustration of how the AL-RNN creates the chaotic dynamics. For the Rössler, trajectories diverge from an unstable spiral point (true position in gray, learned position in black) into the second subregion, where after about half a cycle they are propelled back into the first. For the Lorenz-63, two unstable spiral points (true: gray; learned: black) create the diverging spiraling dynamics in the two lobes, separated by the saddle node in the center. c: Topological graphs of the symbolic coding. While for the Rössler it is fully connected, for the Lorenz-63 the crucial role of the center saddle region in distributing trajectories onto the two lobes is apparent. d: Geometrical divergence ($D_{\text{stsp}}$) among repeated trainings of AL-RNNs ($n=20$), separately evaluated within each subregion, shows close agreement among different training runs. Likewise, low e: normalized distances between fixed point locations and f: relative differences in maximum absolute eigenvalues $\sigma^{\text{max}}$ across $20$ trained models indicate that these topologically minimal representations are robustly identified.

Theorems & Definitions (12)

• Theorem 1
• proof
• Theorem 2
• proof
• Theorem 3
• proof
• Definition 1
• Definition 2
• Definition 3
• proof