Inferring Concepts from Noisy Examples in Hopfield-like Neural Networks

TL;DR

The paper addresses how attractor networks can generalize from limited, noisy examples by inferring archetypal concepts rather than memorizing instances. It introduces a hierarchical two-level pattern structure and a learning rule based on average correlations, analyzed with Replica Symmetric and 1RSB mean-field theory. Key findings include six solution classes, three of which generalize (FS, CR, OE), with rich zero-temperature bifurcation diagrams; 1RSB corrections significantly improve agreement with simulations and raise capacity thresholds. The work demonstrates that a subtle shift in learning rules and pattern structure can enhance generalization while preserving locality, with implications for neural computation and machine learning frameworks dealing with concept extraction from noisy data.

Abstract

We study a variant of the pseudo-inverse learning rule for Hopfield-like Neural Networks, which allows the network to infer archetypal concepts on the basis of a limited number of examples. The mean-field replica theory for this model reveals how this generalization ability is mediated by a multitude of states, with diverse thermodynamic properties, coexisting with the standard Hopfield ones. They appear and vanish through smooth transitions or discontinuous jumps and, interestingly, show much stronger Replica Symmetry Breaking (RSB) effects than the standard Hopfield model, as captured by our 1RSB analysis. Our results, in excellent agreement with numerical simulations, provide deeper insight into the interplay between memory storage and generalization in attractor neural networks.

Figures (3)

• Figure 1: Overlap bifurcation diagrams at $T=0$ and $K=4$. Main figure shows results for $\rho=0.75$, the inset for $\rho=0.8$. The diagrams show different RS MFT solution types characterized by overlaps $m_1$ (dots) and $m_2$ (crosses) as a function of $\alpha$. Red identifies the FS solution, green the CR solution, purple the OE solution.
• Figure 2: Phase boundaries in the temperature-load parameter space for $K=4$ examples per archetype. Main figure shows results for dataset-quality $\rho=0.75$ while the inset for $\rho=0.80$. Red indicates where only the OE exists, Orange indicates where only the FS solution exists, yellow indicates where the CR and the OE solution coexists.
• Figure 3: Comparison between 1RSB mean-field theory predictions and simulations for zero temperature overlap cumulative distribution function at load parameter $\alpha=0.005$ (left); $\alpha=0.015$ (center); $\alpha=0.045$ (right). Other parameters are: $N=8192$, $K=4$ hierarchy levels, correlation strength $\rho=0.75$. Full lines show cumulative distribution functions of overlaps with examples (black lines) and archetype (blue lines). MFT predictions are shown as vertical lines, dashed for overlaps with examples, dotted for overlaps with the archetype. Orange corresponds to the OE solution, red to the CR solution, green to the FS solution, gray to the Hopfield solution. In the right panel a very transparent shade indicates the RS estimate, showcasing how 1RSB improves the matching with simulations.