Prototype Analysis in Hopfield Networks with Hebbian Learning

TL;DR

This work investigates prototype formation in Hopfield networks trained with Hebbian learning, showing that highly correlated subsets of learned states can give rise to unlearned prototype states that dominate attractor spaces and alleviate capacity issues. The authors develop a theoretical stability condition for a representative state formed from a correlated subset, incorporating the number of examples, the noise in those examples, and the count of non-examples, and derive a probability of stability that aligns with classic spin-glass capacity results. They corroborate the theory with Hebbian-Hopfield experiments, demonstrating single and multiple prototype formation, robustness of prototypes to noise, and the dependence of prototype strength on example count and confounding states, as well as energy-profile distinctions between prototypes, learned, and spurious states. Overall, the work links prototype formation to energy landscape characteristics and confirms that prototypes can serve as efficient representatives of large correlated memory sets, with capacity limits consistent with traditional Hopfield results and relevance to broader prototype-learning literature.

Abstract

We discuss prototype formation in the Hopfield network. Typically, Hebbian learning with highly correlated states leads to degraded memory performance. We show this type of learning can lead to prototype formation, where unlearned states emerge as representatives of large correlated subsets of states, alleviating capacity woes. This process has similarities to prototype learning in human cognition. We provide a substantial literature review of prototype learning in associative memories, covering contributions from psychology, statistical physics, and computer science. We analyze prototype formation from a theoretical perspective and derive a stability condition for these states based on the number of examples of the prototype presented for learning, the noise in those examples, and the number of non-example states presented. The stability condition is used to construct a probability of stability for a prototype state as the factors of stability change. We also note similarities to traditional network analysis, allowing us to find a prototype capacity. We corroborate these expectations of prototype formation with experiments using a simple Hopfield network with standard Hebbian learning. We extend our experiments to a Hopfield network trained on data with multiple prototypes and find the network is capable of stabilizing multiple prototypes concurrently. We measure the basins of attraction of the multiple prototype states, finding attractor strength grows with the number of examples and the agreement of examples. We link the stability and dominance of prototype states to the energy profile of these states, particularly when comparing the profile shape to target states or other spurious states.

Figures (20)

• Figure 1: The taxonomy of states in an associative memory. The relationship between stable learned states and attractor states is continuous, measuring the attractor strength. Our work focuses on the distinction between the classes of stable unlearned states; separating prototypes from other spurious states.
• Figure 2: Probability of Error of representative mixture states over ratios of K to N, holding the Bernoulli parameter. The horizontal line at $P_{\text{error}}=0.0036$ is added to indicate the allowable $P_{\text{error}}$ according to Hertz.
• Figure 3: Probability of Error of representative mixture states over ratios of K to N, holding the number of presented examples constant. The horizontal line at $P_{\text{error}}=0.0036$ is added to indicate the allowable $P_{\text{error}}$ according to Hertz.
• Figure 4: Two representative states (top row) with examples constructed by applying increasingly more noise, corresponding to a larger Bernoulli parameter. A larger Bernoulli parameter results in states further from the representative. Yellow corresponds to $1$, purple to $-1$. Note the states are vectors, and the height shown here is for ease of visualization only.
• Figure 5: Plot of the average Manhattan distance between the most recalled state and the representative state. Different plots correspond to different numbers of confounding states, while colors represent different Bernoulli parameters.