On the Expressive Power of Modern Hopfield Networks
Xiaoyu Li, Yuanpeng Li, Yingyu Liang, Zhenmei Shi, Zhao Song
TL;DR
The paper analyzes the expressive power of modern and kernelized Hopfield networks through circuit complexity, proving that poly(n)-precision MHNs with constant depth and O(n) hidden units reside in DLOGTIME-uniform TC0. Consequently, unless TC0 = NC1, such MHNs cannot solve NC1-hard problems like undirected connectivity and tree isomorphism. Kernelized MHNs receive analogous bounds, indicating inherent expressivity limits despite empirical success. These results guide the design of Hopfield-based architectures by clarifying fundamental computational constraints and highlighting gaps between practical performance and worst-case complexity.
Abstract
Modern Hopfield networks (MHNs) have emerged as powerful tools in deep learning, capable of replacing components such as pooling layers, LSTMs, and attention mechanisms. Recent advancements have enhanced their storage capacity, retrieval speed, and error rates. However, the fundamental limits of their computational expressiveness remain unexplored. Understanding the expressive power of MHNs is crucial for optimizing their integration into deep learning architectures. In this work, we establish rigorous theoretical bounds on the computational capabilities of MHNs using circuit complexity theory. Our key contribution is that we show that MHNs are $\mathsf{DLOGTIME}$-uniform $\mathsf{TC}^0$. Hence, unless $\mathsf{TC}^0 = \mathsf{NC}^1$, a $\mathrm{poly}(n)$-precision modern Hopfield networks with a constant number of layers and $O(n)$ hidden dimension cannot solve $\mathsf{NC}^1$-hard problems such as the undirected graph connectivity problem and the tree isomorphism problem. We also extended our results to Kernelized Hopfield Networks. These results demonstrate the limitation in the expressive power of the modern Hopfield networks. Moreover, Our theoretical analysis provides insights to guide the development of new Hopfield-based architectures.