Exponential Dynamic Energy Network for High Capacity Sequence Memory

TL;DR

This work addresses the limitation of traditional energy-based memories in handling sequences by proposing Exponential Dynamic Energy Network (EDEN), a two-timescale architecture that couples a fast high-capacity energy network with a slow modulatory population to drive sequence memory. The authors derive short-timescale energy functions governing local dynamics, obtain an analytic expression for memory-escape times, and show a phase transition between static and dynamic regimes. They demonstrate exponential sequence capacity, $C_{\mathrm{EDEN}} = k(\epsilon,\delta) \, \Big( \frac{e^{\alpha r} e^{\alpha}}{\cosh(\alpha r) \cosh(\alpha)} \Big)^{N-1}$ with $r=\alpha_s/\alpha_c$, highlighting a strong advantage over linear-capacity models, and provide simulations that align with time-cell and ramping-cell phenomena observed in episodic memory. By unifying static and sequential memory under a dynamic energy framework, EDEN offers a scalable, interpretable model for high-capacity temporal memory with potential implications for artificial and biological systems.

Abstract

The energy paradigm, exemplified by Hopfield networks, offers a principled framework for memory in neural systems by interpreting dynamics as descent on an energy surface. While powerful for static associative memories, it falls short in modeling sequential memory, where transitions between memories are essential. We introduce the Exponential Dynamic Energy Network (EDEN), a novel architecture that extends the energy paradigm to temporal domains by evolving the energy function over multiple timescales. EDEN combines a static high-capacity energy network with a slow, asymmetrically interacting modulatory population, enabling robust and controlled memory transitions. We formally derive short-timescale energy functions that govern local dynamics and use them to analytically compute memory escape times, revealing a phase transition between static and dynamic regimes. The analysis of capacity, defined as the number of memories that can be stored with minimal error rate as a function of the dimensions of the state space (number of feature neurons), for EDEN shows that it achieves exponential sequence memory capacity $O(γ^N)$, outperforming the linear capacity $O(N)$ of conventional models. Furthermore, EDEN's dynamics resemble the activity of time and ramping cells observed in the human brain during episodic memory tasks, grounding its biological relevance. By unifying static and sequential memory within a dynamic energy framework, EDEN offers a scalable and interpretable model for high-capacity temporal memory in both artificial and biological systems.

Figures (5)

• Figure 1: Schematic Model and Energy Landscape Behavior of Dynamic Energy Networks: A. The dynamic energy network, EDEN, has asymmetrically interacting slow neurons providing information about the next memory in the sequence to the two fast neural populations. B. Static energy-based networks are used as models of human associative memory where a single memory associated with a provided stimulus is recalled. EDEN, without the slow population, is a static energy network that retrieves a single memory from a collection of stored memories. The system's state ($v_i$), represented by the blue ball, descends the energy surface until a stable memory (energy minimum at state "2") is reached. After retrieval, the state of the system does not change and stays at "2". C. Dynamic energy networks enable associative sequence memory, where the associated memory along with its sequential neighbors are recalled. In EDEN, the energy surface changes in response to the state of the system, causing the minima of the energy surface to change over time (from "2" to "3"), resulting in transitions between memories.
• Figure 2: Simulation of EDEN reveals robust transitions between memory states and the existence of local energy functions: EDEN is simulated to store and retrieve a simple sequence of 5 MNIST digits in numeric order. A The global energy surface with both slow and fast populations of EDEN shows the neural state traversing a valley of the energy surface with occasional energy-increasing regimes. B. The dynamical behavior of the memory overlaps of the fast population ($m^v_{\mu} = \frac{1}{N} \sum_{i=1}^N v_i \xi^{(\mu)}_i$) of EDEN and the analysis of the first principal component (PC1) of the time evolution of its fixed points show the fast population (blue cross) converging to the instantaneous minimum of short-timescale energy functions (red circles). The short-timescale energy minimums are modulated by the slow population. As the slow population approaches the current state of the fast population, the energy minimum switches to the sequentially connected memory. Over time, these short-timescale energy changes slowly so that the fast population has sufficient time to relax at its instantaneous minimum. The long-timescale dynamical behavior of the network can then be assembled from the short-timescale behaviors.
• Figure 3: Escape Time Characteristics of EDEN under different parameter regimes: (left) The analysis of the escape times (in $\mathcal{T}_f$ units) of EDEN under different parameter settings shows two different dynamic regimes. When the coefficient ratio $\alpha_s/\alpha_c > 1$, EDEN has static memories where the dynamic behavior converges to one of the stored memories without any transitions. When the coefficient ratio $\alpha_s/\alpha_c < 1$, EDEN has memory transitions. (right) We take 4 sample cross sections of the phase diagram, shown by the colored horizontal lines on the left. The average time required to escape a memory state is characterized by the timescale ($\mathcal{T}_d/\mathcal{T}_f$) and the coefficient ($\alpha_s/\alpha_c$) ratios. The analytical escape times (the solid lines) computed from the energy function show good agreement with the experimental values (the points) with a mean absolute error of $5.96\mathcal{T}_f$ units.
• Figure 4: Exponential Sequence Memory Capacity of EDEN: The plot shows the fixed point capacity in the $\log_{10}$ scale for EDEN simulated with different $\alpha_c = \alpha$ (with $\alpha_s = 0.999 \, \alpha$) compared with the reference network when small errors ($\delta < 10^{-3}$) are tolerated. The analytic curves are shown as solid lines and experimental values as points. The reference network capacity scales linearly with the asymptotic rate of $O(N)$ (dotted orange line), while EDEN scales exponentially with the asymptotic rate $O(\gamma^N)$ in the number of feature neurons. The exponent base is higher than the limit ($\gamma > 2$), enabling EDEN to reach the available capacity limits of $2^N$ (dotted blue line) in the asymptotic limit of the number of neurons.
• Figure 5: The EDEN neural populations shows behavioral similarity to cells observed in human episodic memory experiments: A The heatmap of the hidden layer neuron activity ordered by time shows time-sensitive behavior analogous to the time cells observed in human episodic memory retrieval experiments of Umbach2020TimeCI. B The slow layer neurons ramp up their activity until it reaches the current memory which in turn induces the transition to the next memory. Rather than an instantaneous drop in their activity, the slow layer slowly ramps down to stabilize the feature layer state on the next memory. This ramp up and ramp down activity is analogous to the activity of ramping cells observed in episodic memory experiments Umbach2020TimeCI.