Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ALADE-SNN: Adaptive Logit Alignment in Dynamically Expandable Spiking Neural Networks for Class Incremental Learning

Wenyao Ni, Jiangrong Shen, Qi Xu, Huajin Tang

TL;DR

This work tackles catastrophic forgetting in class-incremental learning for spiking neural networks by introducing ALADE-SNN, which combines dynamically expandable architectures with adaptive logit alignment and OtoN suppression to balance old and new task representations. The method builds on a DER-style backbone, adding new feature extractors per task while freezing past ones, and trains with a two-stage process that includes representation learning and classifier refinement. Key contributions include identifying logits bias under imbalanced data, proposing adaptive logits alignment to rebalance task logits, and implementing OtoN suppression to prevent old features from hijacking new-task mappings, yielding strong results on CIFAR100 benchmarks and competitive performance on neuromorphic data like DVS-CIFAR10. Together, these techniques advance brain-inspired continual learning in SNNs and point to energy-efficient, real-time processing benefits for neuromorphic systems.

Abstract

Inspired by the human brain's ability to adapt to new tasks without erasing prior knowledge, we develop spiking neural networks (SNNs) with dynamic structures for Class Incremental Learning (CIL). Our comparative experiments reveal that limited datasets introduce biases in logits distributions among tasks. Fixed features from frozen past-task extractors can cause overfitting and hinder the learning of new tasks. To address these challenges, we propose the ALADE-SNN framework, which includes adaptive logit alignment for balanced feature representation and OtoN suppression to manage weights mapping frozen old features to new classes during training, releasing them during fine-tuning. This approach dynamically adjusts the network architecture based on analytical observations, improving feature extraction and balancing performance between new and old tasks. Experiment results show that ALADE-SNN achieves an average incremental accuracy of 75.42 on the CIFAR100-B0 benchmark over 10 incremental steps. ALADE-SNN not only matches the performance of DNN-based methods but also surpasses state-of-the-art SNN-based continual learning algorithms. This advancement enhances continual learning in neuromorphic computing, offering a brain-inspired, energy-efficient solution for real-time data processing.

ALADE-SNN: Adaptive Logit Alignment in Dynamically Expandable Spiking Neural Networks for Class Incremental Learning

TL;DR

This work tackles catastrophic forgetting in class-incremental learning for spiking neural networks by introducing ALADE-SNN, which combines dynamically expandable architectures with adaptive logit alignment and OtoN suppression to balance old and new task representations. The method builds on a DER-style backbone, adding new feature extractors per task while freezing past ones, and trains with a two-stage process that includes representation learning and classifier refinement. Key contributions include identifying logits bias under imbalanced data, proposing adaptive logits alignment to rebalance task logits, and implementing OtoN suppression to prevent old features from hijacking new-task mappings, yielding strong results on CIFAR100 benchmarks and competitive performance on neuromorphic data like DVS-CIFAR10. Together, these techniques advance brain-inspired continual learning in SNNs and point to energy-efficient, real-time processing benefits for neuromorphic systems.

Abstract

Inspired by the human brain's ability to adapt to new tasks without erasing prior knowledge, we develop spiking neural networks (SNNs) with dynamic structures for Class Incremental Learning (CIL). Our comparative experiments reveal that limited datasets introduce biases in logits distributions among tasks. Fixed features from frozen past-task extractors can cause overfitting and hinder the learning of new tasks. To address these challenges, we propose the ALADE-SNN framework, which includes adaptive logit alignment for balanced feature representation and OtoN suppression to manage weights mapping frozen old features to new classes during training, releasing them during fine-tuning. This approach dynamically adjusts the network architecture based on analytical observations, improving feature extraction and balancing performance between new and old tasks. Experiment results show that ALADE-SNN achieves an average incremental accuracy of 75.42 on the CIFAR100-B0 benchmark over 10 incremental steps. ALADE-SNN not only matches the performance of DNN-based methods but also surpasses state-of-the-art SNN-based continual learning algorithms. This advancement enhances continual learning in neuromorphic computing, offering a brain-inspired, energy-efficient solution for real-time data processing.

Paper Structure

This paper contains 10 sections, 8 equations, 3 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Analysis of the Feature Representation
  4. Dynamically Expandable SNNs with Adaptive Logits Alignment
  5. Results
  6. Conclusion
  7. Acknowledgments
  8. Supplemenary Related works
  9. Supplemenary Preliminary
  10. Supplemenary Experimental settings

Figures (3)

  • Figure 1: The overview of the proposed ALADE-SNN. This SNN-based continual learning framework includes two stages of training: the representation learning stage (upper part), where we implement OtoN suppression and knowledge distillation, and classifier learning stage (bottom left part), where we implement adaptive logit align to re-balance the knowledge distribution.
  • Figure 2: The architecture of the framework in comparative experiments. In these experiments, we primarily examine the impact of logits and weight on network performance. (A) The illustration of the overall network structure and its parameter properties. (B) The plot of the mapping relation between the separated deep features and the separated output logits.
  • Figure 3: (Left) The plot of the curve of average accuracy (averaged across all steps) of old classes and new classes under different experimental settings in CIFAR100-B0 with 10 steps. (Right) The corresponding curve of absolute difference.