Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neuro-mimetic Task-free Unsupervised Online Learning with Continual Self-Organizing Maps

Hitesh Vaidya, Travis Desell, Ankur Mali, Alexander Ororbia

TL;DR

This work investigates catastrophic forgetting in online, continual unsupervised learning by evaluating a classical self-organizing map (SOM) and introducing the Continual Kohonen Map (CSOM). CSOM augments SOM with per-unit running variance ($oldsymbol{M}^{oldsymbol{ ho^2}}$), unit-specific learning rates and radii, and a masking mechanism to bound updates, enabling online learning without task boundaries. The authors provide convergence proofs for the variance and weight updates, establish a dynamic equilibrium that balances plasticity and stability, and demonstrate strong empirical performance across MNIST, Fashion-MNIST, KMNIST, and CIFAR-10 in class- and domain-incremental setups, often surpassing baselines and achieving state-of-the-art results in online unsupervised CIFAR-10. The results suggest CSOM as a scalable, memory-efficient approach for lifelong learning in resource-constrained, unsupervised or minimally supervised settings, with potential for integration into supervised and semi-supervised systems and memory-augmented architectures.

Abstract

An intelligent system capable of continual learning is one that can process and extract knowledge from potentially infinitely long streams of pattern vectors. The major challenge that makes crafting such a system difficult is known as catastrophic forgetting - an agent, such as one based on artificial neural networks (ANNs), struggles to retain previously acquired knowledge when learning from new samples. Furthermore, ensuring that knowledge is preserved for previous tasks becomes more challenging when input is not supplemented with task boundary information. Although forgetting in the context of ANNs has been studied extensively, there still exists far less work investigating it in terms of unsupervised architectures such as the venerable self-organizing map (SOM), a neural model often used in clustering and dimensionality reduction. While the internal mechanisms of SOMs could, in principle, yield sparse representations that improve memory retention, we observe that, when a fixed-size SOM processes continuous data streams, it experiences concept drift. In light of this, we propose a generalization of the SOM, the continual SOM (CSOM), which is capable of online unsupervised learning under a low memory budget. Our results, on benchmarks including MNIST, Kuzushiji-MNIST, and Fashion-MNIST, show almost a two times increase in accuracy, and CIFAR-10 demonstrates a state-of-the-art result when tested on (online) unsupervised class incremental learning setting.

Neuro-mimetic Task-free Unsupervised Online Learning with Continual Self-Organizing Maps

TL;DR

This work investigates catastrophic forgetting in online, continual unsupervised learning by evaluating a classical self-organizing map (SOM) and introducing the Continual Kohonen Map (CSOM). CSOM augments SOM with per-unit running variance (), unit-specific learning rates and radii, and a masking mechanism to bound updates, enabling online learning without task boundaries. The authors provide convergence proofs for the variance and weight updates, establish a dynamic equilibrium that balances plasticity and stability, and demonstrate strong empirical performance across MNIST, Fashion-MNIST, KMNIST, and CIFAR-10 in class- and domain-incremental setups, often surpassing baselines and achieving state-of-the-art results in online unsupervised CIFAR-10. The results suggest CSOM as a scalable, memory-efficient approach for lifelong learning in resource-constrained, unsupervised or minimally supervised settings, with potential for integration into supervised and semi-supervised systems and memory-augmented architectures.

Abstract

An intelligent system capable of continual learning is one that can process and extract knowledge from potentially infinitely long streams of pattern vectors. The major challenge that makes crafting such a system difficult is known as catastrophic forgetting - an agent, such as one based on artificial neural networks (ANNs), struggles to retain previously acquired knowledge when learning from new samples. Furthermore, ensuring that knowledge is preserved for previous tasks becomes more challenging when input is not supplemented with task boundary information. Although forgetting in the context of ANNs has been studied extensively, there still exists far less work investigating it in terms of unsupervised architectures such as the venerable self-organizing map (SOM), a neural model often used in clustering and dimensionality reduction. While the internal mechanisms of SOMs could, in principle, yield sparse representations that improve memory retention, we observe that, when a fixed-size SOM processes continuous data streams, it experiences concept drift. In light of this, we propose a generalization of the SOM, the continual SOM (CSOM), which is capable of online unsupervised learning under a low memory budget. Our results, on benchmarks including MNIST, Kuzushiji-MNIST, and Fashion-MNIST, show almost a two times increase in accuracy, and CIFAR-10 demonstrates a state-of-the-art result when tested on (online) unsupervised class incremental learning setting.
Paper Structure (33 sections, 4 theorems, 12 equations, 4 figures, 23 tables, 2 algorithms)

This paper contains 33 sections, 4 theorems, 12 equations, 4 figures, 23 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. The Kohonen Map
  5. The Continual Kohonen Map
  6. Mathematical Analysis
  7. Proof:
  8. Proof:
  9. Proof:
  10. Proof:
  11. Experiments
  12. Datasets
  13. Split MNIST, Fashion MNIST, and KMNIST:
  14. Split CIFAR-10:
  15. Model Baselines:
  16. ...and 18 more sections

Key Result

Lemma 4.1

The series for $\omega_i^2(t)$ converges under the assumption that the sequence of squared deviations ${(\mathbf{x}(t) - \mathbf{M}_i(t))^2}$ is bounded.

Figures (4)

  • Figure 1: Illustration of the overall process for best-matching unit (BMU) selection and synaptic updating of the CSOM for unsupervised online learning.
  • Figure 2: Unit centric parameter update in CSOM inspired from competitive learning
  • Figure 3: Class-incrementally adapted classical SOM (top row) versus continual SOM (bottom row) on: MNIST (Left), FashionMNIST (Middle), and KMNIST (Right).
  • Figure 4: (Left) CSOM containing 100x100 neurons/units trained class incrementally on grayscale images of CIFAR-10. (Right) Two snapshots of trained CSOM clusters

Theorems & Definitions (4)

  • Lemma 4.1
  • Proposition 4.2
  • Theorem 4.3
  • Theorem 4.4