Statistical mechanics of vector Hopfield network near and above saturation

TL;DR

We introduce a vector Hopfield model with $d$-dimensional spins and analyze its equilibrium and out-of-equilibrium behavior. Using replica-symmetric theory, we map the phase diagram as a function of storage $α$ and temperature $T$, finding that the retrieval region shrinks with increasing $d$ and that the zero-temperature capacity scales as $α_c(0) ∝ 1/d$, while transient one-step retrieval or denoising scales as $\tilde{α} ∝ d$. A detailed Hessian analysis reveals a gapless lower edge and a pseudogap with localized soft modes for memory states, contrasting with delocalized modes in spurious states, and the soft-mode structure is tied to the local-field distribution and Onsager reaction. Dynamically, above saturation the network exhibits transient denoising at the first step, with Mattis states showing enhanced one-step retrieval that survives up to $\tilde{α} ∝ d$, consistent with a mean-field glass interpretation where memory states are stable glasses. The results establish connections between high-dimensional vector spin memories, random-matrix theory, and glassy dynamics, with implications for understanding attention-like mechanisms in modern neural architectures.

Abstract

We study analytically and numerically a Hopfield fully-connected network with $d$-dimensional vector spins. These networks are models of associative memory that generalize the standard Hopfield with Ising spins, where $P$ examples are stored in a network of $N$ units as local minima in an energy landscape. We study the equilibrium and out-of-equilibrium properties of the system, considering the system in its retrieval phase $α<α_c$ and beyond, where $α=P/N$ is the capacity of the system and $α_c$ is its critical value, above which storage fails. We derive the Replica Symmetric solution for the equilibrium thermodynamics of the system, together with its phase diagram: we find that the retrieval phase of the network shrinks with growing spin dimension, having ultimately a vanishing critical capacity $α_c\propto 1/d$ in the large $d$ limit. As a trade-off, we observe that in the same limit vector Hopfield networks are able to denoise corrupted input patterns in the first step of retrieval dynamics, up to very large capacities $α\propto d$. We also study the static properties of the system at zero temperature, considering the statistical properties of soft modes of the energy Hessian spectrum. We find that local minima of the energy landscape related to memory states have ungapped spectra with rare soft eigenmodes: these excitations are localized, their measure condensating on the noisiest neurons of the memory state.

Figures (13)

• Figure 1: The model is a genuine Hopfield associative memory: Solution of the $d=3$ VHM \ref{['eq:ModelHam']} in the RS approximation. We show the Mattis magnetization $m$ (left), the linear susceptibility $\chi$ (center) and the free energy density (right) as functions of the capacity $\alpha$, for temperatures $T=0., 0.25, 0.5$. The behavior is analogous to that of the SHM: for any $T$, the solution exists up to a critical capacity $\alpha_c$ and undergoes an equilibrium first order phase transition at $0<\alpha_m<\alpha_c$. We indicate equilibrium solution with continuous lines, while dashed opaque lines identify metastable solutions. Here we show data for spins with norm $\sigma=\sqrt{d}$, thus the free energy density is normalized by a factor $d$ and $0<\chi<1$.
• Figure 2: The retrieval phase shrinks with growing spin dimension: Phase diagrams of VHMs with $d=3, 5, 10, 50$, compared with that of the SHM ($d=1)$. Spins are chosen with norm $\sigma=\sqrt{d}$ in order to fix the no-retrieval spin glass temperature at $\alpha=0$ to unit. Left: the phase diagram is shown in the $T,\alpha$ plane. Retrieval line shrink for growing $d$, while the spin glass line for no-retrieval states converge to unit. In the inset, the zero-temperature critical capacity $\alpha_c(0)$ is shown to decrease as $1/d$. Right: the same phase diagram but with $\alpha"=\alpha d$ on the $x$-axis. Retrieval lines in this scaling regime seem to converge to non-trivial curves. In particular, we find that the critical capacity $\alpha_c(0)\rightarrow 4/27$ in the large $d$ limit, as shown in the inset.
• Figure 3: The pseudogap in the spectral density controls the stability of the system. The spectral density obtained from the solution of eq. \ref{['eq:self_consistent_equation_resolvent']} for the $d=3$ VHM, compared with empirical spectral densities computed from numerical measures. Left: the spectral density for Mattis states. With different dashed lines we show the theoretical predictons for capacities $\alpha=0.02, 0.04$, while continuous lines of different colors detailed in the legend are numerical measures for sizes $N=1000, 2000, 4000$. In the inset, we zoom on the non-trivial bulk close to the lower edge of the spectrum. The remainder of the spectral density has a shape that reminds of that of $P_\eta(\eta)$, as one can fully assess by comparing it with fig. \ref{['fig:distribution_local_fields']} in Appendix \ref{['sec:local_fields_pdf_study']}. There is a good agreement between theory and measures, with visible finite size effects only at low eigenvalues. In particular, while for capacity $\alpha=0.02$ the power tail close to $\lambda=0$ is not captured, for capacity $\alpha=0.04$ we see that the lower tail of the spectrum progressively approach the asymptotic curve for increasing sizes. Right: same but for spurious spin-glass states. Here we only compare theory and data for $\alpha=0.04$, to ease graph legibility. One can see systematic errors between the RS theory and numerical measures similar to those found in figure \ref{['fig:distribution_local_fields']} in Appnedix \ref{['sec:local_fields_pdf_study']}. In addition, the $\rho(\lambda)$ obtained from \ref{['eq:self_consistent_equation_resolvent']} using the RS local fields distribution for no-retrieval states has an unphysical gap close to the lower edge: the numerical spectral density instead is clearly ungapped and seems to approach $\lambda=0$ as $\rho_{emp}(\lambda)\sim \sqrt{\lambda}$, consistently with one expects for the spectral density of spin glass states.
• Figure 4: Soft modes of memory states are localized.. The rescaled IPR $\kappa=N I$ of the $d=3$ VHM, for Mattis states (top panel) and Spin glass states (bottom panel). Top: figure a) compares the theoretical prediction \ref{['eq:rescaled_IPR_theory']} with empirical measures based on eqs \ref{['eq:IPR']}. In the bulk there is an excellent agreement between theory and numerics, while at the spectral edges deviations consistent with eigenvectors localization appear. Figure b) shows the sample-averaged IPR of the softest mode for several values of $\alpha$ detailed in the legend. For increasing sizes, the IPR is not vanishing, validating the localization hypothesis. Bottom: same but for no-retrieval states. Figure c) shows the comparison between theoretical rescaled IPR and numerics: again, the RS theory is not correct since it predicts a non-physical spectral gap. The empirical rescaled IPR seems to be regular for $\lambda\rightarrow 0$. In figure d) we show the sample-averaged IPR of the smallest mode: this quantity is decreasing for increasing sizes in a range $N=\mathcal{O}(10^2):\mathcal{O}(10^3)$.
• Figure 5: Soft modes of Mattis states localize on noisy spins. In this figure we show our measures of correlation for respectively local fields $\eta_i$ and rescaled weights $u_i=N|\bm{\psi}_i|^2$ and local noises $\ell_i$ and rescaled weights, for the $d=3$ VHM. We show heatmaps in panels a) and c) and scatter plots in figures b) and d). Superimposed to scatter plots we show with dark continuous lines typical curves of $\eta$ and $\ell$ at fixed $u$, computed respectively as $\eta_{typ}(u)=\exp\overline{\frac{1}{N}\sum_i\log (\eta_i)\delta(u-u_i)}$ and $\ell_{typ}(u)=\exp\overline{\frac{1}{N}\sum_i \log(\ell_i)\delta(u-u_i)}$. The dashed purple line marks the position of the sample-averaged local noise $\overline{\ell}=\frac{1-m}{2}$, while black dashed lines mark respectively the empirical lower and upper bounds of $\ell_{typ}$. Sizes are $N=250, 500, 1500$, capacity is $\alpha=0.04$. We considered a number of samples per size such that $N_s=1500\times\frac{100}{N}$.