Transient dynamics of associative memory models

Abstract

Associative memory models such as the Hopfield network and its dense generalizations with higher-order interactions exhibit a "blackout catastrophe" -- a discontinuous transition where stable memory states abruptly vanish when the number of stored patterns exceeds a critical capacity. This transition is often interpreted as rendering networks unusable beyond capacity limits. We argue that this interpretation is largely an artifact of the equilibrium perspective. We derive dynamical mean-field equations for graded-activity dense associative memory models, with the Hopfield model as a special case, using a bipartite cavity approach. We solve the resulting self-consistent equations using an iterative numerical scheme. We show that patterns can be transiently retrieved with high accuracy above capacity despite the absence of stable attractors. This occurs because slow regions persist in the above-capacity energy landscape near stored patterns as lingering traces of the stable basins that existed below capacity. The same transient-retrieval effect occurs in below-capacity networks initialized outside basins of attraction. "Transient-recovery curves" provide a concise visual summary of these effects, revealing graceful, non-catastrophic changes in retrieval behavior above capacity and allowing us to compare the behavior across interaction orders. This dynamical perspective reveals energy landscape structure obscured by equilibrium analysis, including slow regions near stored patterns that persist above capacity, and suggests biological neural circuits may exploit transient dynamics for memory retrieval. Furthermore, our approach suggests ways of understanding computational properties of neural circuits without reference to fixed points and yields new theoretical results on generalizations of the Hopfield model.

Figures (7)

• Figure 1: Schematics of the dense associative memory model. (a) Neuronal formulation with higher-order interactions. Nodes represent neurons and black dots indicate connections (tensor elements $T_{i j_1 \cdots j_n}$). (b) Equivalent formulation as a bipartite network with neurons $x_i(t)$ and overlaps $m^\mu(t)$ connected in a bipartite manner through stored patterns $\xi_i^\mu$. (c) Schematic of bipartite cavity scheme used to derive the DMFT.
• Figure 2: Dynamical evolution of order parameters for Hopfield ($n=1$) and dense associative memory models ($n=2,4$). (a) Below-capacity dynamics with $\alpha = 0.10, 0.05, 0.001$ for $n=1,2,4$, respectively. (b) Above-capacity dynamics with $\alpha = 0.20, 0.10, 0.005$ for $n=1,2,4$, respectively. In (a) and (b), we show (top) raw overlap $m(t)$, (middle) equal-time correlation $C^\phi(t,t)$, and (bottom) normalized overlap $\bar{m}(t) = m(t)/(\sigma_\xi\sqrt{C^\phi(t,t)})$. Gray traces show individual finite-size simulations with $N = 20000, 2000, 200$ for $n = 1, 2, 4$, respectively; black lines show simulation medians; and magenta lines show DMFT predictions.
• Figure 3: Transient-recovery curves for Hopfield model ($n=1$) and dense associative memory models ($n=2,4$). Each curve plots the maximum normalized overlap $\bar{m}_{\text{max}}$ achieved during dynamical evolution versus the initial normalized overlap $\bar{m}_{\text{init}}$. Different curves within each panel correspond to different memory loads $\alpha = P/N^n$. The diagonal line is the trivial lower bound where maximum overlap equals initial overlap.
• Figure 4: Energy dynamics for different initial overlaps ($g = 1.5$). (a) Below-capacity dynamics with $\alpha < \alpha_c$. (b) Above-capacity dynamics with $\alpha > \alpha_c$. In (a) and (b), columns show $n=1, 2, 4$ from left to right. Different curves in each panel correspond to different initial normalized overlaps $\bar{m}_{\text{init}}$. Gray traces show individual finite-size simulations; black lines show simulation medians; and magenta lines show DMFT predictions. (c) Example showing correspondence between energy decay and overlap evolution for $n=1$, $\alpha=0.16$. Top: normalized overlap $\bar{m}(t)$; middle: energy $\varepsilon(t)$; bottom: energy derivative $d\varepsilon(t)/dt$. Horizontal axis shows time on log scale. The fast rise and slow decay of the normalized overlap correspond to fast decay and slow decay of the energy, consistent with the system navigating shallow energy landscape features near stored patterns that eventually drive it away from the memory. (d) Schematic of the energy landscape structure near a stored pattern in the below-capacity regime, where a stable basin exists (top), and in the above-capacity regime, where a slow, unstable region persists despite the elimination of the stable basin (bottom).
• Figure 5: Optimal readout time $t_{\text{opt}} = \mathop{\mathrm{arg\,max}}\limits_t \bar{m}(t)$ as a function of initial normalized overlap $\bar{m}_{\text{init}}$. Columns show $n=1, 2, 4$ from left to right. Rows show increasing values of $\alpha$. Magenta lines show DMFT values of $t_{\text{opt}}$. Green shaded regions indicate where the network enters a stable retrieval state, making the optimal readout time undefined since any late time works equally well.