Data-Driven Reduced Modeling of Recurrent Neural Networks

TL;DR

The paper introduces data-driven Spectral Submanifolds (SSMs) as robust, low-dimensional invariants attached to fixed points in high-dimensional recurrent neural networks (RNNs). By leveraging the SSMLearn algorithm, it derives polynomial reduced-order models that capture core nonlinear dynamics across tasks, including context-dependent decisions, input-controlled oscillations, and memory-based behavior. The results demonstrate one- and two-dimensional slow/unstable manifolds that reproduce full RNN trajectories and reveal the dynamical motifs underlying line attractors, ring attractors, and heteroclinic memory structures, even in the presence of perturbations or parameter changes. This framework provides precise, interpretable descriptions of neural computations and offers a general, noise-tolerant approach for analyzing complex RNN dynamics beyond traditional time-scale separation assumptions.

Abstract

Artificial Recurrent Neural Networks (RNNs) are widely used in neuroscience to model the collective activity of neurons during behavioral tasks. The high dimensionality of their parameter and activity spaces, however, often make it challenging to infer and interpret the fundamental features of their dynamics. In this study, we employ recent nonlinear dynamical system techniques to uncover the core dynamics of several RNNs used in contemporary neuroscience. Specifically, using a data-driven approach, we identify Spectral Submanifolds (SSMs), i.e., low-dimensional attracting invariant manifolds tangent to the eigenspaces of fixed points. The internal dynamics of SSMs serve as nonlinear models that reduce the dimensionality of the full RNNs by orders of magnitude. Through low-dimensional, SSM-reduced models, we give mathematically precise definitions of line and ring attractors, which are intuitive concepts commonly used to explain decision-making and working memory. The new level of understanding of RNNs obtained from SSM reduction enables the interpretation of mathematically well-defined and robust structures in neuronal dynamics, leading to novel predictions about the neural computations underlying behavior.

Figures (11)

• Figure 1: SSMs carrying the RNN reduced dynamics. (Upper left) Unstable manifold at order five in coordinates $(\eta_1, y_2)$, where $\eta_1$ parametrizes the spectral subspace $E_1$ and $y_i = x_i-x_{i,0}$ are the original RNN coordinates centered around the unstable fixed point. We depict a full-order test trajectory and the corresponding reduced trajectory on the SSM (after $t\approx\frac{1}{\lambda_2}$, when transients have decayed, purple dot.) converging to the fixed point with the larger domain of attraction along the unstable manifold. Black dots correspond to initial conditions. The fixed points in the plot (crosses) are the fixed points of the full model, lying on the unstable manifold and correctly reproduced by the reduced-order model. The Manifold Fitting Error (MFE), the mean distance between observed trajectories and trajectories projected on the manifold, has magnitude $\approx 0.007$, and the Normal Mean Trajectory Error (NMTE), the mean reconstruction error of the reduced model, is $\approx 0.03$. (Upper right) The slowest SSM at order three and test full-order and reduced-order trajectories plotted in coordinates $(\eta_1, y_2)$, with $y_i$ now centered around the stable fixed point. The MTE and NMTE have magnitude $\approx 0.05$. (Center) Graphs of the right-hand sides of the reduced-order models: on the 1D unstable manifold (left) and on the 1D slowest SSM (right), both at order five. On the left plot, the unstable fixed point (red cross) serves as a boundary for the domains of attraction of the two stable fixed points (blue crosses) on the unstable manifold. On the right plot, we observe convergence to the stable fixed point. (Bottom) Sketch of the robust (normally hyperbolic) 2D invariant manifold (slow manifold) in the extended phase space constructed including the $s_2$-parameter direction (left), in coordinates $(\eta_1, s_2, y_1)$, where $\eta_1$ and $y_1$ are as above, and in coordinates $(\eta_1, y_1)$ for selected $s_2$ values (right). We see how the 1D latent dynamics underlying flexible decision-making change when varying the relevant sensory input $s_2$ from negative to positive values, passing through 0, resulting in a change in the preferred choice for the network. When $s_2=0$, there is a stable fixed point that attracts initial conditions around zero and, locally, the slow manifold coincides with its 1D slowest SSM (in round brackets).
• Figure 2: One-dimensional slow manifolds (unstable manifolds) of the system for different values of the sensory input parameter $s_2$ (selected by context 2). The fixed point (coloured crosses) the network has learned to converge to depends on the sign of $s_2$. The way the RNN realizes this input-output relationship is changing location to the unstable fixed point (central cross) along the slow manifold depending on $s_2$: the unstable fixed point separates the domains of attraction of the two stable fixed points in the reduced dynamics, so its location (dashed line) determines the asymptotic behavior of trajectories, given their initial conditions. The crosses correspond to the fixed points, the two stable one divided by the unstable one. We display the arrows indicating effect of the input vector on the reduced dynamics (projection onto the SSM). This picture shows that the projection of the input vector (or its mean, in the noisy case) on the reduced dynamics does not determine the asymptotic behavior (choice) of the RNN, as we can see that, for example, it would lead to the wrong direction for $s_2= 0.036$ (red arrows). However, whenever the perturbed initial conditions lie within the correct domain of attraction, the convergence to the correct choice is guaranteed (see also \ref{['appC']}).
• Figure 3: SSM carrying the core RNN dynamics and parameter-dependent SSMs with their reduced dynamics projected to the spectral subspaces. (Upper left) Unstable manifold at order three in coordinates $(\eta_1, \eta_2,y_5)$, where $(\eta_1, \eta_2)$ are the coordinates of the unstable subspace. The stable limit cycle on the unstable manifold is globally attracting: in the picture, one full and one reduced trajectory are converging to the stable limit cycle. The MFE and the NMTE calculated on test trajectories have magnitude $\approx 0.02$ and $\approx 0.5$, respectively. (Bottom left ) Projections onto the unstable subspace of the streamlines of the right-hand side of the reduced-order model on the two-dimensional unstable manifold up to order six, with projected full and reduced trajectories. (Upper right) Two-dimensional unstable manifolds for different values of the parameter $u$ (and corresponding output frequency values $f$) plotted in the three-dimensional space with coordinates $(\eta_1, \eta_2,y_5)$. The manifolds are shifted on the $y_5$-axis for visualization purposes. (Bottom right) Projection onto the unstable subspace of the streamlines of the right-hand side of the reduced-order model on the two-dimensional unstable manifold up to order six for parameter $u$ corresponding to a frequency of $3.9$ Hz, with projected full and reduced trajectories.
• Figure 4: The heteroclinic orbit on the 2D SSM. (Upper left) The two-dimensional unstable manifold attached to the unstable fixed point carries the reduced dynamics. Trajectories (full model trajectories in purple and reduced model in orange) disposed on a circular curve converge to the stable fixed point in infinite time. (Bottom left) Norm of the right-hand side of the reduced model on the unstable manifold plotted on the unstable subspace with coordinates $(\eta_1, \eta_2)$, with darker blues corresponding to larger norm values. The MFE and the NMTE calculated on test trajectories have magnitude $10^{-2}$ and $10^{-1}$, respectively. (Bottom right) Heteroclinic orbits connecting the unstable fixed point on the bottom right and the stable fixed point on the upper left, projected onto the unstable subspace with coordinates $(\eta_1,\eta_2)$. Streamlines of the right-hand side of the reduced model are projected onto the unstable subspace (in gray). The structure is normally hyperbolic, as we can see from the directions of the slow and the fast eigenvectors of the linearization around the stable fixed point of the full model. (Upper right) Pictorial representation of the normally hyperbolic heteroclinic structure, as an example of normally hyperbolic invariant manifold in Fenichel. The unstable manifold of the unstable fixed point Q coincides with the slow stable manifold of the stable fixed point P, and the rate of normal attraction to P is bigger than the rate of tangential compression.
• Figure 5: Pictorial representation of the context-dependent task performed by the RNN in mante_context-dependent_2013 and renate (right). Pictorial representation of the sine-wave generator RNN in renate (left).