Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamical Properties of Dense Associative Memory

Kazushi Mimura, Jun'ichi Takeuchi, Yuto Sumikawa, Yoshiyuki Kabashima, Anthony C. C. Coolen

TL;DR

This work addresses the dynamical properties of dense associative memory, a high-capacity variant of modern Hopfield networks, by applying generating functional analysis (GFA) to derive an exact large-N description of retrieval dynamics. The authors show that, for n ≥ 3, the dynamics are governed by macroscopic order parameters (overlap, correlation, and response) with a retarded self-interaction that suppresses self-induced noise, enabling robust recall and larger basins of attraction. They provide explicit dynamical equations and demonstrate quantitative agreement with numerical simulations, establishing a framework to evaluate convergence time and storage capacity and offering a general methodology for energy-based models beyond dense associative memory. The results have implications for designing robust memory architectures and can inform the development of memory-augmented systems and modern Hopfield networks.

Abstract

Dense associative memory, a fundamental instance of modern Hopfield networks, can store a large number of memory patterns as equilibrium states of recurrent networks. While the stationary-state storage capacity has been investigated, its dynamical properties have not yet been discussed. In this paper, we analyze the dynamics using an exact approach based on generating functional analysis. We show results on convergence properties of memory retrieval, such as the convergence time and the size of the attraction basins. Our analysis enables a quantitative evaluation of the convergence time and the storage capacity of dense associative memory, which is useful for model design. Unlike the traditional Hopfield model, the retrieval of a pattern does not act as additional noise to itself, suggesting that the structure of modern networks makes recall more robust. Furthermore, the methodology addressed here can be applied to other energy-based models, and thus has the potential to contribute to the design of future architectures.

Dynamical Properties of Dense Associative Memory

TL;DR

This work addresses the dynamical properties of dense associative memory, a high-capacity variant of modern Hopfield networks, by applying generating functional analysis (GFA) to derive an exact large-N description of retrieval dynamics. The authors show that, for n ≥ 3, the dynamics are governed by macroscopic order parameters (overlap, correlation, and response) with a retarded self-interaction that suppresses self-induced noise, enabling robust recall and larger basins of attraction. They provide explicit dynamical equations and demonstrate quantitative agreement with numerical simulations, establishing a framework to evaluate convergence time and storage capacity and offering a general methodology for energy-based models beyond dense associative memory. The results have implications for designing robust memory architectures and can inform the development of memory-augmented systems and modern Hopfield networks.

Abstract

Dense associative memory, a fundamental instance of modern Hopfield networks, can store a large number of memory patterns as equilibrium states of recurrent networks. While the stationary-state storage capacity has been investigated, its dynamical properties have not yet been discussed. In this paper, we analyze the dynamics using an exact approach based on generating functional analysis. We show results on convergence properties of memory retrieval, such as the convergence time and the size of the attraction basins. Our analysis enables a quantitative evaluation of the convergence time and the storage capacity of dense associative memory, which is useful for model design. Unlike the traditional Hopfield model, the retrieval of a pattern does not act as additional noise to itself, suggesting that the structure of modern networks makes recall more robust. Furthermore, the methodology addressed here can be applied to other energy-based models, and thus has the potential to contribute to the design of future architectures.

Paper Structure

This paper contains 19 sections, 3 theorems, 40 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Background
  3. Contributions
  4. Related Works
  5. Preliminalies
  6. Notations
  7. Dense Associative Memory
  8. Outline of Generating Functional Analysis
  9. Analysis
  10. Path Probability
  11. Generating Functional
  12. Main results
  13. Discussion
  14. Numerical Analysis and Computer Simulations
  15. Connection to Related Analyses
  16. ...and 4 more sections

Key Result

Lemma 1

By averaging over the memory patterns, the generating functional is given by where where $A(\ell,k) = \bigl( \!\! \!\! \bigr)^2 \; k! \; B(\ell-k)^2$, and $B(m) = \boldsymbol{1}_{m\hbox{:even}} \; (m-1)!!.$

Figures (1)

  • Figure 1: Recalling process of Krotov's dense associative memory with $F(x)=x^n$ and $n=3$. Left: computer simulations, 100 trials, $N=512$. Right: theory. (a) $\alpha_3'=0.1$. (b) $\alpha_3'=0.2$. (c) $\alpha_3'=0.3$. (d) $\alpha_3'=0.4$.

Theorems & Definitions (4)

  • Definition 1
  • Lemma 1
  • Proposition 1
  • Corollary 1