Supervised Hebbian Learning

TL;DR

The paper introduces a supervised Hebbian learning framework that lets a Hopfield network infer archetypes from noisy examples and maps its performance across dataset quality, size, and noise via a phase-diagram informed by statistical mechanics. It shows that, for structureless data, the supervised Hopfield model is equivalent to a Restricted Boltzmann Machine, providing an interpretable training route and unifying biological and artificial learning perspectives. Extending to structured data, the work reveals quasi-ultrametric organization and replica-symmetry-breaking signatures, motivating a replica-hierarchy (1RSB) hidden layer that significantly enhances MNIST-type classification. The results establish quantitative links between dataset properties, learning dynamics, and architectural depth, offering a principled path to bridging shallow Hebbian learning with deeper, structured representations in neural networks.

Abstract

In neural network's Literature, Hebbian learning traditionally refers to the procedure by which the Hopfield model and its generalizations store archetypes (i.e., definite patterns that are experienced just once to form the synaptic matrix). However, the term "Learning" in Machine Learning refers to the ability of the machine to extract features from the supplied dataset (e.g., made of blurred examples of these archetypes), in order to make its own representation of the unavailable archetypes. Here, given a sample of examples, we define a supervised learning protocol by which the Hopfield network can infer the archetypes, and we detect the correct control parameters (including size and quality of the dataset) to depict a phase diagram for the system performance. We also prove that, for structureless datasets, the Hopfield model equipped with this supervised learning rule is equivalent to a restricted Boltzmann machine and this suggests an optimal and interpretable training routine. Finally, this approach is generalized to structured datasets: we highlight a quasi-ultrametric organization (reminiscent of replica-symmetry-breaking) in the analyzed datasets and, consequently, we introduce an additional "replica hidden layer" for its (partial) disentanglement, which is shown to improve MNIST classification from 75% to 95%, and to offer a new perspective on deep architectures.

Figures (12)

• Figure 1: Behaviour of the supervised HNN as the control parameter are varied. Panel $a$: phase diagram highlighting the ergodic (E), the spin-glass (SG) and the retrieval (R) phase versus $T$ and $\alpha$; the transition line between the SG phase and the R phase depends on $\rho$ and three cases are shown: $\rho =0$ (dashed line, corresponding to AGS theory), $\rho=0.1$ (dashed-dotted line), and $\rho=0.2$ (dotted line). Panel $b$: critical load $\alpha_c$ obtained for $T=0$ and as a function of $\rho$. Panel $c$: Estimate of the Mattis magnetization versus $\rho$ by MC simulations for systems of size $N=5000$; different loads are considered and plotted in different colors (brighter nuances correspond to larger values of $\alpha$, as reported on the right); the vertical lines represent the transition points as predicted by statistical mechanics. Panel $d$: From data presented in panel $c$ we derive the susceptibility w.r.t. $\rho$ and notice that the peaks approximately match the transition points (by a finite-size scaling we checked that the match gets closer as $N$ is made larger).
• Figure 1: Comparison between signal-to-noise predictions (solid lines) and MC simulations (symbols) for the unsupervised (dark color, $\circ$) and the supervised (bright color, $\square$) Hebbian learning in the noiseless $\beta \to \infty$ limit. More precisely, the theoretical estimates are obtained by eq. \ref{['astronzi']} and eq. \ref{['2stability']}, while the numerical estimates are obtained by fixing a certain number $M$ of examples and determining the minimal value of the dataset quality $r$ such that the mean overlap between the neural configuration $\boldsymbol \sigma$ and a chosen archetype, say $\boldsymbol \xi^1$, i.e., the magnetization $\langle m \rangle$, is (approximately) unitary, the operation is then repeated for several values of $M$ ranging exponentially from $2$ to $2^{12}$. Note that the theoretical estimates perfectly interpolate the numerical outcomes.
• Figure 2: Comparison between HNN and RBM performances. Panel $a$: we fix a certain value for the expected magnetization $\langle m \rangle$ and we derive from Eq. \ref{['eq:sce_m']}, obtained theoretically for the HNN, how $r$ and $M$ should be tuned in order to retain this value constant (solid line); an analogous analysis is repeated numerically for the RBM where now $m$ is evaluated as the overlap between the visible layer and a given archetype (symbols); different values of magnetization are considered and represented with different symbols. Panel $b$: Expected value of the RBM magnetization versus the training time and for given values of $r$ and $M$, under on-line contrastive divergence (CD-1) Hinton1MC; the long-time value corresponds to the theoretical estimate obtained for the HNN for the same choice of $r$ and $M$ (horizontal lines). Panel $c$: we sampled $1.5\times 10^4$ couples $(\alpha, \rho) \in (0,0.2) \times (0, 0.5)$ by Sobol's low-discrepancy sequence; for each extraction (represented by a cross in the inset) we build a RBM of size $N=5000$ and $K = \alpha N$, we generate a set $\mathcal{S}$ of examples and we set the machine weights as $\boldsymbol W = \bar{\boldsymbol \eta}$. Then, we initialize the visible layer as a test example $\tilde{\boldsymbol \eta}^{\nu}$, we run MC simulations and we evaluate $\langle z_{\nu} \rangle$, whose histogram is depicted in the main plot, distinguishing between cases inside (blue) and outside (grey) the retrieval region.
• Figure 2: Signal-to-Noise analysis. Left panel: Contour plot for the one-step-MC magnetization $m^{(2)}$ defined in \ref{['magnet']} and evaluated by MC simulations, run at different values of $\alpha$ and $\rho$; the solid line corresponds to the curve $\alpha\left(1+\rho\right)^{2}+\rho=1$ obtained by eq. \ref{['eq:stability_sup']} by setting as tolerance level $\theta=\frac{1}{\sqrt{2}}$. Right panel: Contour plot for the one-step-MC magnetization $m^{(2)}$ defined in \ref{['magnet']} and evaluated by MC simulations, run at different values of $r$ and $M$ (notice the log-scale); the solid lines are again obtained by eq. \ref{['eq:stability_sup']} but now plotted as a function of $(r,\log_2M)$ and we see that, as the quality $|r|$ reaches zero, more and more examples are needed to ensure the retrieval of the archetype. The colormap on the right is shared by the two panels.
• Figure 3: Evidence of RSB in structured datasets. Upper Plots: we compare the empirical overlap distribution $\mathcal{P}(q)$ obtained for the random (panel $a$), the MNIST (panel $b$), and the fashion-MNIST (panel $c$) datasets; three different item sizes are also considered, see the legend. From left to right, we move from a RS scenario where $\mathcal{P}(q)$ exhibits two peaks that get sharper as the item size increases, to a RSB scenario where $\mathcal{P}(q)$ is bimodal but with increasing broadness as the item size increases. Lower plots: we report the violation of the Ghirlanda-Guerra identities ($GG_1$, $GG_2$) and the violation of self-averaging $SA$ as obtained for the random (panel $d$), the MNIST (panel $e$) and the fashion-MNIST (panel $f$) datasets. Again from left to right we move from a replica-symmetric scenario where the self-averaging relations hold and the Ghirlanda-Guerra relations (corresponding to trivial identities) are fast vanishing, to a picture resembling broken replica symmetry, where self-averaging does not hold any longer but the Ghirlanda-Guerra relations are still preserved (this time in a not trivial manner). See the SM SM for further explanation.

Theorems & Definitions (42)

• Definition 1
• Definition 2
• Definition 3
• Remark 1
• Theorem 1
• Proposition 1
• proof
• Proposition 2
• proof
• Remark 2