Unraveling the Hidden Dynamical Structure in Recurrent Neural Policies

TL;DR

The paper investigates why recurrent policies generalize well under partial observability by analyzing their hidden dynamics through a dynamical-systems lens. It shows that fully optimized recurrent policies converge to attracting limit cycles in the joint agent-environment state $x_t=(s_t,h_t)$, and that episodic resets act as a periodic drive under a Periodically-Kicked Drive (PKD) mechanism to stabilize these cycles. A key contribution is the demonstration of a structural isomorphism between neural limit cycles and behavioral trajectories, revealed via Behavioral Potential Fields and Canonical Correlation Analysis, with causal evidence from counterfactual injections and action-consistency verification. The work provides a unified account linking memory organization, robustness to environmental variability, and relational behavioral structure, with implications for both artificial and biological motor control systems. Overall, the findings suggest that neural manifolds shaped by limit cycles encode task-relational geometry that underpins adaptive behavior across diverse tasks, architectures, and learning protocols.

Abstract

Recurrent neural policies are widely used in partially observable control and meta-RL tasks. Their abilities to maintain internal memory and adapt quickly to unseen scenarios have offered them unparalleled performance when compared to non-recurrent counterparts. However, until today, the underlying mechanisms for their superior generalization and robustness performance remain poorly understood. In this study, by analyzing the hidden state domain of recurrent policies learned over a diverse set of training methods, model architectures, and tasks, we find that stable cyclic structures consistently emerge during interaction with the environment. Such cyclic structures share a remarkable similarity with \textit{limit cycles} in dynamical system analysis, if we consider the policy and the environment as a joint hybrid dynamical system. Moreover, we uncover that the geometry of such limit cycles also has a structured correspondence with the policies' behaviors. These findings offer new perspectives to explain many nice properties of recurrent policies: the emergence of limit cycles stabilizes both the policies' internal memory and the task-relevant environmental states, while suppressing nuisance variability arising from environmental uncertainty; the geometry of limit cycles also encodes relational structures of behaviors, facilitating easier skill adaptation when facing non-stationary environments.

Figures (23)

• Figure 1: Illustration of the Hybrid Dynamical System (HDS) framework.(a) Unified State Space. The execution state $x_t = (s_t, h_t)$ couples physical coordinates (orange axes, $\mathcal{S}$) and neural hidden states (purple axes, $\mathcal{H}$) into a single high-dimensional product space. (b) Example of a Limit Cycle. The system converges to a stable periodic orbit $\Gamma$. The vector field visualizes the basin of attraction, showing how trajectories from diverse initial conditions rapidly contract onto this cycle.
• Figure 2: Overview of the experimental framework. We systematically evaluate recurrent policy dynamics across three axes of variation: (1) Task Families, ranging from partially observed grid maze navigation to high-dimensional Procgen games; (2) Training Pipelines, comparing gradient-based policy optimization method (PPO) against gradient-free evolution strategies; and (3) Recurrent Architectures, spanning classic RNNs, gated recurrent units (GRUs), and modern state-space model architecture Mamba.
• Figure 3: Task-adapted policies stabilize into limit cycles.(a) In the stable regime, the physical trajectory forms a topological loop enabled by episodic resets, while neural memory traces a low-dimensional closed loop (PCA projection). (b) Across task families and instances, the recurrent hidden-state trajectory converges to a stable closed orbit with task-dependent shape.
• Figure 4: Robustness and recovery under external perturbations. The cycle acts as an attractor: after an external perturbation, execution transiently deviates but rapidly re-synchronizes with the nominal limit cycle. Additional experimental results on perturbation are provided in Appendix \ref{['app:perturb']}.
• Figure 5: Universality of contractive dynamics across architectures, tasks, and algorithms. FTLI distributions (computed with a horizon $K=1000$) for paired rollouts under small perturbations across three settings: (a) GRU/ES on Mazes; (b) GRU/PPO on Jumper; (c) Mamba/PPO on Jumper. Optimized policies (blue) consistently show $\lambda_{FTLI} < 0$ (contraction), while random networks (yellow) show $\lambda_{FTLI} > 0$ (chaos).

Theorems & Definitions (8)

• Definition 4.1: Periodic Input Drive
• Definition 4.2: Contractive Region
• Definition 4.3: Dissipative Policy Dynamics (Restricted)
• Definition 4.4: Stroboscopic Map
• Lemma 4.5: Contraction of the Stroboscopic Map
• proof
• Theorem 4.6: Existence of a Stable Limit Cycle within a Contractive Region
• proof