Rethinking Hebbian Principle: Low-Dimensional Structural Projection for Unsupervised Learning
Shikuang Deng, Jiayuan Zhang, Yuhang Wu, Ting Chen, Shi Gu
TL;DR
This work rethinks Hebbian learning by casting it as an optimal low-dimensional projection problem and introduces SPHeRe, a purely feedforward, Hebbian-inspired unsupervised framework. SPHeRe uses a lightweight auxiliary projection to preserve the input's structural relationships via a loss on Gram matrices, coupled with an orthogonality constraint to decorrelate features. The authors prove that minimizing the SPHeRe loss corresponds to a principal subspace projection and demonstrate state-of-the-art performance among Hebbian-inspired unsupervised methods on CIFAR-10/100 and Tiny-ImageNet, with strong continual and transfer learning and robust reconstruction results. The approach shows that biologically motivated learning rules can be competitive within modern deep learning when augmented with efficient auxiliary modules, offering a scalable alternative to backpropagation for unsupervised pre-training.
Abstract
Hebbian learning is a biological principle that intuitively describes how neurons adapt their connections through repeated stimuli. However, when applied to machine learning, it suffers serious issues due to the unconstrained updates of the connections and the lack of accounting for feedback mediation. Such shortcomings limit its effective scaling to complex network architectures and tasks. To this end, here we introduce the Structural Projection Hebbian Representation (SPHeRe), a novel unsupervised learning method that integrates orthogonality and structural information preservation through a local auxiliary nonlinear block. The loss for structural information preservation backpropagates to the input through an auxiliary lightweight projection that conceptually serves as feedback mediation while the orthogonality constraints account for the boundedness of updating magnitude. Extensive experimental results show that SPHeRe achieves SOTA performance among unsupervised synaptic plasticity approaches on standard image classification benchmarks, including CIFAR-10, CIFAR-100, and Tiny-ImageNet. Furthermore, the method exhibits strong effectiveness in continual learning and transfer learning scenarios, and image reconstruction tasks show the robustness and generalizability of the extracted features. This work demonstrates the competitiveness and potential of Hebbian unsupervised learning rules within modern deep learning frameworks, demonstrating the possibility of efficient and biologically inspired learning algorithms without the strong dependence on strict backpropagation. Our code is available at https://github.com/brain-intelligence-lab/SPHeRe.