On Computational Limits of Modern Hopfield Models: A Fine-Grained Complexity Analysis
Jerry Yao-Chieh Hu, Thomas Lin, Zhao Song, Han Liu
TL;DR
The paper addresses the computational limits of memory retrieval in modern Hopfield models by framing retrieval as an Approximate Nearest Neighbor Search problem and applying fine-grained reductions under the Strong Exponential Time Hypothesis. It reveals a norm-based phase transition that delineates when sub-quadratic time methods are possible, and shows a complementary hardness regime. It then constructs an almost linear-time modern Hopfield model via low-rank polynomial approximation, providing retrieval error bounds and confirming exponential memory capacity. The work offers theoretical insights for scalable Hopfield-driven components in large foundation models and transformer architectures, guiding future efforts toward efficient, high-capacity associative memories.
Abstract
We investigate the computational limits of the memory retrieval dynamics of modern Hopfield models from the fine-grained complexity analysis. Our key contribution is the characterization of a phase transition behavior in the efficiency of all possible modern Hopfield models based on the norm of patterns. Specifically, we establish an upper bound criterion for the norm of input query patterns and memory patterns. Only below this criterion, sub-quadratic (efficient) variants of the modern Hopfield model exist, assuming the Strong Exponential Time Hypothesis (SETH). To showcase our theory, we provide a formal example of efficient constructions of modern Hopfield models using low-rank approximation when the efficient criterion holds. This includes a derivation of a lower bound on the computational time, scaling linearly with $\max\{$# of stored memory patterns, length of input query sequence$\}$. In addition, we prove its memory retrieval error bound and exponential memory capacity.