The dimensionality of the Hopfield model

TL;DR

This work introduces the Binary Intrinsic Dimension (BID) as a global geometric measure tailored for binary data and applies it to the Hopfield network to probe phase structure and transitions. By fitting a linear dimensionality model to the distribution of Hamming distances and relating distances to overlaps via $x=1-\frac{2}{N}r$, the authors extract BID as $\hat{\theta}_0$ and connect BID to the overlap distribution $q$ through $\mathbb{E}_\theta[x] \approx q$, offering a geometry-based perspective on retrieval, spin-glass, and paramagnetic phases. The results show that BID scales linearly with system size $N$ in retrieval and paramagnetic phases, but exhibits sublinear scaling in the spin-glass phase, signaling strong correlations, with a Gaussian approximation linking BID to overlap moments and yielding a scaling relation $\gamma_{\mathrm{Std}}+\gamma_{\mathrm{BID}} \approx 1/2$. BID proves robust to finite-size effects and sampling schemes, providing a practical, unsupervised tool to detect phase transitions and to study the relationship between state-space geometry and standard order parameters. The study lays groundwork for applying BID to more complex networks and for exploring non-linear extensions of the dimensionality model, with implications for understanding binary neural systems and related architectures.

Abstract

We use the Binary Intrinsic Dimension (BID), a geometrical measure designed for binary data, to analyze the Hopfield model, a paradigmatic spin system from statistical mechanics, machine learning and neuroscience. The BID allows us to characterize the phases and transitions of this system, and moreover it is robust against finite-size effects that interfere with the correct numerical estimation of the spin-glass order parameter ($q$). We observe that the BID scales linearly with system size in the retrieval and paramagnetic phases, where the correlations between spins are small, and exhibits sublinear scaling in the whole spin-glass phase, highlighting its correlated structure. Furthermore, we establish a direct relationship between the BID and the overlap distribution, unveiling a novel connection between the geometry of the state-space and standard spin order parameters.

Figures (11)

• Figure 1: (a) Binary Dataset. (b) Illustration of Hamming distance between $\bm s$ and $\bm s'$ as the length of the shortest path connecting both spins in the $N$ dimensional hypercube. (c) The BID is obtained by fitting the model \ref{['eq:bid_model']} to the empirical distribution of Hamming distances $\{r(\bm s,\bm s'): \forall \bm s\neq\bm s' \in \mathcal{S}\}$.
• Figure 2: (a) Mapping of the parameter space using the BID for two initial magnetization values $m_0 = 0.0$ and $m_0=0.9$. The transition temperatures are indicated by their corresponding labels. Vertical and horizontal dashed lines represent cuts at fixed $\alpha$ and $T$, respectively. (b) Numerical Hopfield order parameters $(\widehat{q},\widehat{m})$ computed at fixed value of $\alpha$ and $T$ alongside the $BID$, normalized per bit. The vertical dashed lines correspond to the transition points identified in panel (a), with different line-styles indicating the transitions. Simulations were performed with $N=1024$, $N_S=2500$, averages are computed over $30$ realizations.
• Figure 3: Magnetization with the different patterns during asynchronous dynamics. For each temperature, the traces of several $m^\nu(t)$ are shown along with the corresponding histogram of distances. In the corner of each trace plot we show the numerical order parameters $(\hat{q},\hat{m})$ estimated by \ref{['eq:order-params-estimates']}. The value of $\hat{q}$ is also shown in the histograms as the dotted vertical line. Simulation with $N=1024$ and $\alpha=0.04$. Note that in the spin glass phase, the model fit in black dashed lines is accurate for the right hand side of the probability distribution containing larger overlap values (intra-cluster distances). See \ref{['sec:BID-and-q']} and \ref{['subsec:range_R']} for further details.
• Figure 4: (a) Typical histograms at temperature $T=1.0$ for different system sizes showing the bimodality as a finite size effect. The last panel shows in gray the estimated $\hat{q}$ as a function of $N$ and in teal the expected value of $x$ computed with the model \ref{['eq:bid_model']} after fitting. Error bars correspond to the standard deviation across 20 independent network realizations. The thermodynamic value $q \approx 0.17$ is shown as the horizontal dashed line. In the histograms the vertical lines correspond to the values of $\hat{q}$ and $\mathbb{E}_\theta[x]$ in gray and teal color respectively. (b) Scaling of the order parameters with $N$ for different temperatures. The temperatures are the same as in figure \ref{['fig:mag-hist']} except for $T=1.2$ which is the critical temperature $T_g$ for $\alpha=0.04$. The curve of $\hat{q}$ for $T=1.0$ is the same one displayed on the last panel of (a). All simulations are done with $\alpha=0.04$ and $m_0=0.9$.
• Figure 5: (a) Typical histograms at two temperatures for various $N$. The last panel shows the standard deviation of the distribution as a function of $N$ for each temperature along with the exponent $\gamma$ in $\text{Std}_\theta[x]\sim N^{-\gamma}$. (b) Scaling exponents for the BID and $\text{Std}_\theta\left[x\right]$ as a function of temperature. To obtain the value of $\gamma$ and its dispersion we performed linear regression with errors on the values $\frac{\text{BID}}{N}$, $\text{Std}_{\theta}[x]$ vs. $N$ for $2^{10} \leq N \leq 2^{14}$. Experiments with $\alpha=0.04$.