SNAP: Stopping Catastrophic Forgetting in Hebbian Learning with Sigmoidal Neuronal Adaptive Plasticity

TL;DR

Sigmoidal Neuronal Adaptive Plasticity (SNAP) is introduced an artificial approximation to Long-Term Potentiation for ANNs by having the weights follow a sigmoidal growth behaviour allowing the weights to consolidate and stabilize when they reach sufficiently large or small values.

Abstract

Artificial Neural Networks (ANNs) suffer from catastrophic forgetting, where the learning of new tasks causes the catastrophic forgetting of old tasks. Existing Machine Learning (ML) algorithms, including those using Stochastic Gradient Descent (SGD) and Hebbian Learning typically update their weights linearly with experience i.e., independently of their current strength. This contrasts with biological neurons, which at intermediate strengths are very plastic, but consolidate with Long-Term Potentiation (LTP) once they reach a certain strength. We hypothesize this mechanism might help mitigate catastrophic forgetting. We introduce Sigmoidal Neuronal Adaptive Plasticity (SNAP) an artificial approximation to Long-Term Potentiation for ANNs by having the weights follow a sigmoidal growth behaviour allowing the weights to consolidate and stabilize when they reach sufficiently large or small values. We then compare SNAP to linear weight growth and exponential weight growth and see that SNAP completely prevents the forgetting of previous tasks for Hebbian Learning but not for SGD-base learning.

Figures (11)

• Figure 1: Illustrations of different weight growth behaviours
• Figure 2: Accuracy of Hebbian MLP on i.i.d. datasets MNIST and FashionMNIST as a function of lateral inhibition hyperparameter $\lambda$ and the type of weight growth. Legend: the first word is the weight growth type of the hidden layer and the second word is the weight growth type of the output layer. So that for example, red lines represent sigmoidal weight growth for the hidden layer and linear weight growth for the output layer. The solid lines denote neuron-wise weight growth while the dotted lines denote synapse-wise weight growth (cf. table \ref{['tab:growth-eqns']}).
• Figure 3: Accuracy of Hebbian MLP on the sequential task versions of MNIST and FashionMNIST as a function of lateral inhibition hyperparameter $\lambda$ and the type of weight growth. Legend: the first word is the weight growth type of the hidden layer and the second word is the weight growth type of the output layer. So that for example, red lines represent sigmoidal weight growth for the hidden layer and linear weight growth for the output layer. The solid lines denote neuron-wise weight growth while the dotted lines denote synapse-wise weight growth (cf. table \ref{['tab:growth-eqns']}).
• Figure 4: Comparison of top-performing Hebbian models on sequential MNIST task learning experiment with various hidden and classification layer weight growth types. Legend: Each line represents a specific class's test accuracy. For instance, the blue line represents the test accuracy of MNIST classes 0 and 1 throughout the experiment.
• Figure 5: Simplified Model Configuration