Input-Driven Dynamics for Robust Memory Retrieval in Hopfield Networks

TL;DR

The paper addresses how external inputs can be leveraged to robustly retrieve memories in Hopfield networks by reframing retrieval as an online, input-driven process. It introduces the Input-Driven Plasticity (IDP) Hopfield model, where the input modulates synaptic weights via $W(u)$ and reshapes the energy landscape $E(x;u)$ to create a memory hierarchy. The authors derive existence and stability thresholds, connect IDP to modern Hopfield architectures, and show that noise and mixed inputs drive retrieval toward the deepest energy well, enabling rapid, noise-assisted memory transitions and glitch correction. This approach offers a biologically plausible mechanism for continual learning and links traditional attractor networks to diffusion-based analyses and transformer-inspired modern Hopfield formulations, with potential impact on both neuroscience and machine learning.

Abstract

The Hopfield model provides a mathematically idealized yet insightful framework for understanding the mechanisms of memory storage and retrieval in the human brain. This model has inspired four decades of extensive research on learning and retrieval dynamics, capacity estimates, and sequential transitions among memories. Notably, the role and impact of external inputs has been largely underexplored, from their effects on neural dynamics to how they facilitate effective memory retrieval. To bridge this gap, we propose a novel dynamical system framework in which the external input directly influences the neural synapses and shapes the energy landscape of the Hopfield model. This plasticity-based mechanism provides a clear energetic interpretation of the memory retrieval process and proves effective at correctly classifying highly mixed inputs. Furthermore, we integrate this model within the framework of modern Hopfield architectures, using this connection to elucidate how current and past information are combined during the retrieval process. Finally, we embed both the classic and the new model in an environment disrupted by noise and compare their robustness during memory retrieval.

Figures (12)

• Figure 1: Comparison between classic Hopfield and IDP Hopfield models. (a) A slowly morphing sequence of noisy images is presented as a input to the observer, who updates its belief state to retrieve the memory closest to the current image $u$. This adaptation process occurs continuously. (b) In the classic model, the network state is set to an initial condition $x(0)$ equal to the current image $u$ and then the Hopfield dynamics performs the memory retrieval task. (c) In the proposed input-driven plasticity model, the network initial condition is arbitrary, the image $u$ modifies the synaptic weights $W(u)$, and the Hopfield dynamics with modified synaptic weights performs the memory retrieval task. This dynamics is well posed and naturally tracks the morphing images also when the image is time-varying $u=u(t)$. (d) In the classic model, the Hopfield dynamics is a gradient descent for the energy $\mathrm{E}(x;W)$: the blue ball, representing the neural state, rolls from an initial condition towards a stable minimum point (cat memory). Therefore, the retrieval process is successful when the noisy image $x(0)=u$ (dotted cat) lies in the region of attraction of the correct memory (cat memory). (e) In the proposed model, the noisy image $u$ directly modifies the synaptic matrix $W(u)$ and in turn the energy landscape $\mathrm{E}(x;W(u))$, thereby extending the region of attraction of the correct memory. The retrieval process is successful from generic initial conditions when the correct memory is the unique minimum of the landscape. Specifically, in the panel, the noisy image (dotted cat) renders the correct memory a minimum (cat memory) and the incorrect memory (dog memory) no longer an equilibrium of $\mathrm{E}(x;W(u))$.
• Figure 2: Energy landscapes for IDP Hopfield model for varying saliency weights. Stable and unstable equilibria are depicted as green stars and brown dots, respectively. Recall the existence threshold is $\alpha_{\textup{existence}}=1$ and, when multiple memories exist in the input, the stability threshold satisfies $\alpha_{\textup{stability}}>1$. (a) "no memories" $\alpha_1<1$, $\alpha_2<1$: when no memory is sufficiently strong in the input, the only global minimum is at the origin and it is globally attractive for the dynamics. This situation corresponds to a confusion state for the network, in which the input is not strong enough to evoke any retrieval. (b) "one memory" $\alpha_1<1$, $\alpha_2>1$: when the saliency weight for precisely one memory is above $\alpha_{\textup{existence}}$, two symmetric equilibria appear corresponding to the memory and they are attractive from almost all initial conditions. In this case, the origin is a saddle point. (c) "one stable and one unstable memory" $\alpha_2>\alpha_{\textup{stability}}>\alpha_1>1$: two symmetric equilibria appear corresponding to the stable memory and they are attractive from almost all initial conditions. The other two symmetric equilibria are saddle points and the origin becomes an unstable maximum. (d) "two stable memories" $\alpha_{2}>\alpha_{1}>\alpha_{\textup{stability}}$: four symmetric stable equilibria appear, corresponding to the two memories of the model. The memory associated to the dominant saliency weight carves deeper valley in the energy landscape than the other memory. When $\alpha_{1}\approx\alpha_{\textup{stability}}$, the shallowness of the valley associated to the first memory facilitates outward jumps due to stochastic fluctuations.
• Figure 3: Illustration of the modern Hopfield reformulation \ref{['featl']},\ref{['memol']},\ref{['alphal']} for the IDP Hopfield model with input filter. The symbol $\odot$ denotes a Hadamard entrywise product. (a) Neural network representation with interconnected layers and synaptic weights $M_\alpha$, $M_x$, and $M_y$. (b) Block diagram representation, where each block $\frac{1}{1+\tau s}$, where $\tau=\tau_\alpha,\tau_x,\tau_y$, denotes a low-pass filter with cutoff frequency $1/\tau$, each block $M_\alpha$, $M_x$, and $M_y$ denotes a matrix-vector multiplication, and each block $\Psi_{\alpha}(\cdot)$, $\Psi_{x}(\cdot)$, and $\Psi_{y}(\cdot)$ denotes an activation function.
• Figure 4: Stationary probability densities $\mathbb{P}_{\infty}(x)$ (plotted as surface for $N=2$) that are equilibrium solutions of the F.P.E. \ref{['FPE']}. (a): stationary probability density when two memories have equal saliency weights ($\alpha_1=\alpha_2=3/2$). In this case, the IDP Hopfield model is equivalent to the classic Hopfield model. (b): stationary probability density when a single memory is dominant ($\alpha_2=9/4>\alpha_1=3/2$). In this case, the memory associated to the dominant saliency weight absorbs most of the probability density and thus enforces a stochastic confinement of the trajectories.
• Figure 5: Intersections of the activation function $\psi(\gamma)=\tanh(\gamma)$ with the dissipation line $\frac{\gamma}{\alpha}$ for different values of $\alpha$. The dissipation lines where the parameter $\alpha\leq{1}$ have just one intersection with the chosen activation function. Instead, only for the value $\alpha=2>1$ we obtain two intersections between the dissipation line and the activation function in the semipositive interval $\mathbb{R}_{+}$.

Theorems & Definitions (10)

• Theorem 1: Existence of retrievable memories
• proof
• Theorem 2: Local stability of equilibria and memories
• proof
• Theorem 3: Critical saliency $\alpha^{\star}$
• proof
• Definition 1: Energy of the IDP Hopfield model
• Theorem 4: Energy wells
• proof
• Definition 2