Saturation Self-Organizing Map

TL;DR

SatSOM addresses catastrophic forgetting in continual learning by introducing a neuron-specific saturation mechanism that decays $\lambda_i$ and $\sigma_i$ as neurons accumulate information, effectively freezing saturated units and freeing capacity for new knowledge. The approach extends Self-Organizing Maps with per-neuron plasticity control, BMU-based training, and a quantile-based, sparse inference that emphasizes active regions of the map. Empirical results on FashionMNIST and KMNIST show SatSOM achieves memory retention comparable to kNN without storing past data and outperforms OnlineEWC, with ablations confirming the saturation mechanism as the primary contributor. The work offers a lightweight, interpretable alternative for continual learning and suggests promising paths for extending saturation concepts to deeper architectures and on-device learning.

Abstract

Continual learning poses a fundamental challenge for neural systems, which often suffer from catastrophic forgetting when exposed to sequential tasks. Self-Organizing Maps (SOMs), despite their interpretability and efficiency, are not immune to this issue. In this paper, we introduce Saturation Self-Organizing Maps (SatSOM)-an extension of SOMs designed to improve knowledge retention in continual learning scenarios. SatSOM incorporates a novel saturation mechanism that gradually reduces the learning rate and neighborhood radius of neurons as they accumulate information. This effectively freezes well-trained neurons and redirects learning to underutilized areas of the map.

Figures (18)

• Figure 1: Symbolic visualization of SatSOM inference on the FashionMNIST dataset. An input image is passed into the model. For each neuron its activation is calculated based on its similarity to the image. The label values stored in the activated neurons are aggregated into the final class probabilities.
• Figure 2: This is an example visualization on a SatSOM trained to classify the FashionMNIST dataset. The prototypes are visualized as little images that are later compared with the new input to perform classification. They have been tinted based on the class that has the highest probability in their respected label-prototype.
• Figure 3: Visualization of a SatSOM trained on the KMNIST dataset, annotated with key mathematical elements.
• Figure 4: Mean accuracy on all FashionMNIST classes through the 10 phases. SatSOM memory retention is comparable to that of the kNN. OnlineEWC results do not surpass the 30% threshold.
• Figure 5: Mean accuracy on all KMNIST classes through the 10 phases. There is a significant increase in standard deviation after the introduction class 8.