iMixer: hierarchical Hopfield network implies an invertible, implicit and iterative MLP-Mixer
Toshihiro Ota, Masato Taki
TL;DR
This work posits a Hopfield–MLP-Mixer correspondence and introduces iMixer, a hierarchical Hopfield-based generalization of the MLP-Mixer in which the token-mixing module is invertible and propagates from the output back to the input via an implicit, iteratively solved i-Res block. By deriving iMixer from a three-layer hierarchical Hopfield network and implementing a fixed-point approximation, the authors enable end-to-end training with standard optimization. Empirically, iMixer matches or slightly outperforms vanilla MLP-Mixer on CIFAR-10 and remains competitive across CIFAR-100, Food-101, ImageNet-1k, and Stanford Cars, with performance improvements tied to model size and spectral normalization stability. The results suggest that viewing MetaFormers through the lens of Hopfield dynamics provides a principled path for designing and understanding Transformer-like architectures, with potential extensions to deeper hierarchies and other vision tasks.
Abstract
In the last few years, the success of Transformers in computer vision has stimulated the discovery of many alternative models that compete with Transformers, such as the MLP-Mixer. Despite their weak inductive bias, these models have achieved performance comparable to well-studied convolutional neural networks. Recent studies on modern Hopfield networks suggest the correspondence between certain energy-based associative memory models and Transformers or MLP-Mixer, and shed some light on the theoretical background of the Transformer-type architectures design. In this paper, we generalize the correspondence to the recently introduced hierarchical Hopfield network, and find iMixer, a novel generalization of MLP-Mixer model. Unlike ordinary feedforward neural networks, iMixer involves MLP layers that propagate forward from the output side to the input side. We characterize the module as an example of invertible, implicit, and iterative mixing module. We evaluate the model performance with various datasets on image classification tasks, and find that iMixer, despite its unique architecture, exhibits stable learning capabilities and achieves performance comparable to or better than the baseline vanilla MLP-Mixer. The results imply that the correspondence between the Hopfield networks and the Mixer models serves as a principle for understanding a broader class of Transformer-like architecture designs.