Smooth embeddings in contracting recurrent networks driven by regular dynamics: A synthesis for neural representation
Vikas N. O'Reilly-Shah, Alessandro Maria Selvitella
TL;DR
This work provides a mathematical synthesis showing that contracting recurrent networks driven by regular dynamics on circles or tori implement $C^1$ embeddings of the driving dynamics via generalized synchronization. By treating the invariant manifold as the base system and enforcing mild bunching, the authors derive embedding guarantees with dimension bounds $N>2d$, which are biologically plausible for cortical populations. The framework integrates generalized synchronization, delay-coordinate embedding, and reservoir computing, yielding concrete expectations about when prediction-trained recurrent circuits form smooth, topology-preserving latent representations that encode recent input history. It clarifies how temporal structure in neural representations arises and offers a principled baseline for comparing regular versus chaotic dynamical regimes in neural computation and perception, with direct relevance to the State Space Theory of Consciousness. The synthesis also outlines practical implications for experiments and simulations, and identifies key open questions around training dynamics, plasticity, and extensions beyond ESP.
Abstract
Recurrent neural networks trained for time-series prediction often develop latent trajectories that preserve qualitative structure of the dynamical systems generating their inputs. Recent empirical work has documented topology-preserving latent organization in trained recurrent models, and recent theoretical results in reservoir computing establish conditions under which the synchronization map is an embedding. Here we synthesize these threads into a unified account of when contracting recurrent networks yield smooth, topology-preserving internal representations for a broad and biologically relevant class of inputs: regular dynamics on invariant circles and tori. Our contribution is an integrated framework that assembles (i) generalized synchronization and embedding guarantees for contracting reservoirs, (ii) regularity mechanisms ensuring differentiability of the synchronization map under mild constraints, and (iii) a base-system viewpoint in which the invariant manifold generating the input stream is treated as the driving system. In this regular setting, the conditions commonly viewed as restrictive in chaotic-attractor analyses become mild and readily satisfied by standard contractive architectures. The framework clarifies how representational content in recurrent circuits is inherently historical: the network state encodes finite windows of input history rather than instantaneous stimuli. By consolidating disparate empirical and theoretical results under common assumptions, the synthesis yields concrete, testable expectations about when prediction-trained recurrent circuits should (or should not) form smooth latent embeddings and how required state dimension scales with the intrinsic dimension of the driving dynamics.