DFORM: Diffeomorphic vector field alignment for assessing dynamics across learned models

TL;DR

DFORM addresses the challenge of comparing dynamics across learned neural models by learning a diffeomorphic coordinate map $H$ that aligns vector fields via a Lie-derivative-based orbital-loss. By using an invertible residual network and a bidirectional training scheme, it yields a continuous orbital similarity measure that captures functional dynamical likeness beyond attractor structure. The framework is demonstrated on canonical nonlinear systems, large RNNs, and memory-task networks, revealing that functionally similar dynamics can be realized by different architectures or coordinate representations. This approach provides a principled, end-to-end method to quantify and interpret dynamical similarity with potential broad impact in neuroscience-inspired modeling and dynamical systems research.

Abstract

Dynamical system models such as Recurrent Neural Networks (RNNs) have become increasingly popular as hypothesis-generating tools in scientific research. Evaluating the dynamics in such networks is key to understanding their learned generative mechanisms. However, comparison of learned dynamics across models is challenging due to their inherent nonlinearity and because a priori there is no enforced equivalence of their coordinate systems. Here, we propose the DFORM (Diffeomorphic vector field alignment for comparing dynamics across learned models) framework. DFORM learns a nonlinear coordinate transformation which provides a continuous, maximally one-to-one mapping between the trajectories of learned models, thus approximating a diffeomorphism between them. The mismatch between DFORM-transformed vector fields defines the orbital similarity between two models, thus providing a generalization of the concepts of smooth orbital and topological equivalence. As an example, we apply DFORM to models trained on a canonical neuroscience task, showing that learned dynamics may be functionally similar, despite overt differences in attractor landscapes.

Figures (8)

• Figure 1: DFORM Schematic. A. Many efforts to compare leaned models resort to dimensionality reduction and assessment of limit sets (e.g., attractors), often by visualization. B. We propose DFORM to learn a diffeomorphism that directly transforms vector fields, thus allowing for rigorous assessment of their similarity.
• Figure 2: Alignment between two-dimensional systems. The DFORM network used here contains 40 instead of 10 layers. Left: vector field (arrows) and simulated trajectories (gray lines) for the first system. Middle: push-forward of the vector field and trajectories by the DFORM model. Right: vector field and trajectories of the second system. Initial conditions for the trajectories were the same as in the middle panel.
• Figure 3: Pairwise similarity between five groups of linear systems. Each group has a different number of positive and negative eigenvalues as indicated in the labels. Each block represents the average similarity over 15 different pairs. Note: the upper and lower triangle are the same.
• Figure 4: Memory task for DFORM analysis. Networks maintain fixation around the origin for a duration of time denoted as $t = T_{Fix}$. Subsequently, a stimulus (highlighted in red) is introduced to the network and remains visible until the time $t = T_{TPres}$. Following this presentation, a variable delay period ensues, concluding with a predetermined response time, during which the network produces a 'saccade' to the intended, context-dependent location (e.g., green vs. blue).
• Figure 5: Similarity between models computed by DFORM and SVCCA. Note: the upper and lower triangle are the same.