Short-term plasticity recalls forgotten memories through a trampoline mechanism

TL;DR

The paper addresses how short-term associative synaptic plasticity interacts with a fixed, structured Hopfield memory to affect retrieval and forgetting. It combines static cavity methods and dynamical mean-field theory (DMFT) to show that plasticity yields only marginal static gains in memory capacity, but can dramatically enhance retrieval dynamics by dynamically reshaping an energy landscape that traps transient activity near stored memories. The work derives DMFT equations with a memory kernel that blends the unlabeled-pattern response and a plasticity kernel, and analyzes the fixed-point stability and an optimal plasticity timescale p that maximizes recall above the static capacity. The findings reveal a trampoline-like mechanism where past activity deforms the neural landscape to aid retrieval, with implications for understanding working memory and designing plastic artificial memories that leverage transient dynamics for robust recall.

Abstract

We analyze continuous Hopfield associative memories augmented by additional, rapid short-term associative synaptic plasticity. Through the cavity method, we determine the boundary between the retrieval and forgetting, or spin-glass phase, of the network as a function of the fraction of stored memories and the neuronal gain. We find that short-term synaptic plasticity yields marginal improvements in critical memory capacity. However, through dynamical mean field theory, backed by extensive numerical simulations, we find that short-term synaptic plasticity has a dramatic impact on memory retrieval above the critical capacity. When short-term synaptic plasticity is turned on, the combined neuronal and synaptic dynamics descends a high-dimensional energy landscape over both neurons and synapses. The energy landscape over neurons alone is thus dynamic, and is lowered in the vicinity of recent neuronal patterns visited by the network, just like the surface of a trampoline is lowered in the vicinity of regions recently visited by a heavy ball. This trampoline-like reactivity of the neuronal energy landscape to short-term plasticity in synapses can lead to the recall of stored memories that would otherwise have been forgotten. This occurs because the dynamics without short-term plasticity transiently moves towards a stored memory before departing away from it. Thus short-term plasticity, operating during the transient, lowers the energy in the vicinity of the stored memory, eventually trapping the combined neuronal and synaptic dynamics at a fixed point close to the stored memory. In this manner, short-term plasticity enables the recall of memories that would otherwise be forgotten, by trapping transients that would otherwise escape. We furthermore find an optimal time constant for short-term synaptic plasticity, matched to the transient dynamics, to empower recall of forgotten memories.

Figures (6)

• Figure 1: Retrieval capabilities of Hopfield recurrent neural network with short-term associative synaptic plasticity. The heatmap shows the long-time overlap of the neuron's output with a target pattern, $m^*_\infty$, when the network is initialized in its vicinity. The long-time overlap is measured using dynamical mean-field theory, for different values of the fraction of stored pattern $\alpha$, and the short-term plasticity strength $k$. The red dashed line denotes the phase boundary, $\alpha_c(k)$, between the retrieval phase of the network and a forgetting phase, as obtained from a static analysis of the network behavior. When $\alpha>\alpha_c(k)$, without synaptic plasticity or for moderate values of $k$, the long time overlap of the network with the target pattern decays to zero. This region is where catastrophic forgetting takes place (white region). However, as the short-term plasticity strength is increased, we report the occurrence of plastic retrieval: long-time recovery of the target pattern takes place even when the network is loaded above its critical capacity, $\alpha>\alpha_c(k)$. The neural gain is set to be $\gamma=3.4$.
• Figure 2: Phase diagram of the fixed point of an Hopfield-RNN with short-term associative synaptic plasticity. Different phases are obtained as the fraction of stored patterns $\alpha$ and the inverse gain $\gamma^{-1}$ are varied. we identify a retrieval phase with a single fixed point (light green region) where the overlap of the network activity with the target pattern is larger than $0$; a multiple-fixed points retrieval phase (dark green region), where the network is prone to nonlinear instabilities but it maintains a finite overlap with the target pattern. A paramagnet phase (blue region), where the network converges to a unique, quiescent, fixed point, with no overlap with the target pattern; and a spin-glass phase (orange region) with multiple fixed points and zero overlap with the target pattern. The phase diagrams are for null, and moderate values of the short-term plasticity strength ($k=0$, panel (a), and $k=0.4$, panel (b), respectively).
• Figure 3: Plastic retrieval stabilizes transient memories Temporal evolution of the overlap $m^*(t)$, as obtained from finite size simulations (dashed lines) and by numerical solution of the DMFT (solid lines). The network is initialized in the vicinity of a target pattern, with the initial overlap set to $m^*(0) \approx 0.46$, and the evolution of $m^*(t)$ for different values of the short-term plasticity strength $k$ is displayed. We have chosen the fraction of stored patterns to be above the critical capacity, $\alpha>\alpha_c(k)$. For moderate value of $k$, the retrieval of the target pattern is only a transient phenomenon. But when the plastic strength becomes larger than a critical threshold, plastic retrieval takes place, and long-time recovery of the target pattern is possible.
• Figure 4: Short-term synaptic plasticity and linear excitations. Eigenvalue spectrum of the Jacobian of the dynamics of the full population of neuron and synapses, at a fixed point of the plastic retrieval regime for $k=1$. The support of the spectrum lies entirely on the negative real axis, and the spectrum is peaked around the theoretical prediction of Sec. \ref{['sec:jacobian']} (dashed blue and red lines).
• Figure 5: Optimal timescale of plastic retrieval. Long time overlap $m_\infty^*$ obtained through plastic retrieval at $\alpha=0.2$, $\gamma=3.4$, and at different values of the short-term plastic strength $k$, as a function of the short-term plasticity timescale $p$. The final overlap exhibits a maximum around $p\approx 2$, consistent with the timescale over which transient retrieval takes place.