Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Sleep-Like Unsupervised Replay Improves Performance when Data are Limited or Unbalanced

Anthony Bazhenov, Pahan Dewasurendra, Giri Krishnan, Jean Erik Delanois

TL;DR

The paper tackles the problem of ANN performance collapse under limited or imbalanced training data by implementing a sleep-like replay mechanism with unsupervised, local Hebbian plasticity. The method, SRC, maps an ANN to an SNN, applies Hebbian updates during a sleep phase driven by Poisson inputs, and then remaps back to an ANN, with optional post-sleep fine-tuning. Results show 20–30% accuracy gains for 0.5–10% data, while larger datasets benefit from post-sleep fine-tuning to maintain performance; Sleep also improves class balance by strengthening critical synapses and producing sparser representations. This work links neuroscience-inspired sleep consolidation to practical ML, offering a data-efficient, unsupervised strategy to enhance memory performance when data are scarce or imbalanced.

Abstract

The performance of artificial neural networks (ANNs) degrades when training data are limited or imbalanced. In contrast, the human brain can learn quickly from just a few examples. Here, we investigated the role of sleep in improving the performance of ANNs trained with limited data on the MNIST and Fashion MNIST datasets. Sleep was implemented as an unsupervised phase with local Hebbian type learning rules. We found a significant boost in accuracy after the sleep phase for models trained with limited data in the range of 0.5-10% of total MNIST or Fashion MNIST datasets. When more than 10% of the total data was used, sleep alone had a slight negative impact on performance, but this was remedied by fine-tuning on the original data. This study sheds light on a potential synaptic weight dynamics strategy employed by the brain during sleep to enhance memory performance when training data are limited or imbalanced.

Sleep-Like Unsupervised Replay Improves Performance when Data are Limited or Unbalanced

TL;DR

The paper tackles the problem of ANN performance collapse under limited or imbalanced training data by implementing a sleep-like replay mechanism with unsupervised, local Hebbian plasticity. The method, SRC, maps an ANN to an SNN, applies Hebbian updates during a sleep phase driven by Poisson inputs, and then remaps back to an ANN, with optional post-sleep fine-tuning. Results show 20–30% accuracy gains for 0.5–10% data, while larger datasets benefit from post-sleep fine-tuning to maintain performance; Sleep also improves class balance by strengthening critical synapses and producing sparser representations. This work links neuroscience-inspired sleep consolidation to practical ML, offering a data-efficient, unsupervised strategy to enhance memory performance when data are scarce or imbalanced.

Abstract

The performance of artificial neural networks (ANNs) degrades when training data are limited or imbalanced. In contrast, the human brain can learn quickly from just a few examples. Here, we investigated the role of sleep in improving the performance of ANNs trained with limited data on the MNIST and Fashion MNIST datasets. Sleep was implemented as an unsupervised phase with local Hebbian type learning rules. We found a significant boost in accuracy after the sleep phase for models trained with limited data in the range of 0.5-10% of total MNIST or Fashion MNIST datasets. When more than 10% of the total data was used, sleep alone had a slight negative impact on performance, but this was remedied by fine-tuning on the original data. This study sheds light on a potential synaptic weight dynamics strategy employed by the brain during sleep to enhance memory performance when training data are limited or imbalanced.
Paper Structure (4 sections, 1 figure)

This paper contains 4 sections, 1 figure.

Table of Contents

  1. Introduction
  2. Algorithm
  3. Results
  4. Acknowledgments

Figures (1)

  • Figure 1: Accuracy on MNIST (A) and FMNIST (B) with mean (lines) and standard deviation (error bars) across 10 trials. X-axis - log of the relative amount of data used for training (e.g., 0.01=1% of data). Blue - baseline (after ANN training); Orange - baseline + sleep; Green - baseline + sleep + fine-tuning. Note significant gain in accuracy after sleep phase on low data. The sleep phase reduced performance on high data but was largely recovered by fine-tuning.