Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast gradient-free activation maximization for neurons in spiking neural networks

Nikita Pospelov, Andrei Chertkov, Maxim Beketov, Ivan Oseledets, Konstantin Anokhin

TL;DR

It is shown that highly selective neurons can form already in the early epochs of training and in the early layers of a convolutional spiking network, which is the first time that effective AM has been applied to SNNs.

Abstract

Elements of neural networks, both biological and artificial, can be described by their selectivity for specific cognitive features. Understanding these features is important for understanding the inner workings of neural networks. For a living system, such as a neuron, whose response to a stimulus is unknown and not differentiable, the only way to reveal these features is through a feedback loop that exposes it to a large set of different stimuli. The properties of these stimuli should be varied iteratively in order to maximize the neuronal response. To utilize this feedback loop for a biological neural network, it is important to run it quickly and efficiently in order to reach the stimuli that maximizes certain neurons' activation with the least number of iterations possible. Here we present a framework with an efficient design for such a loop. We successfully tested it on an artificial spiking neural network (SNN), which is a model that simulates the asynchronous spiking activity of neurons in living brains. Our optimization method for activation maximization is based on the low-rank Tensor Train decomposition of the discrete activation function. The optimization space is the latent parameter space of images generated by SN-GAN or VQ-VAE generative models. To our knowledge, this is the first time that effective AM has been applied to SNNs. We track changes in the optimal stimuli for artificial neurons during training and show that highly selective neurons can form already in the early epochs of training and in the early layers of a convolutional spiking network. This formation of refined optimal stimuli is associated with an increase in classification accuracy. Some neurons, especially in the deeper layers, may gradually change the concepts they are selective for during learning, potentially explaining their importance for model performance.

Fast gradient-free activation maximization for neurons in spiking neural networks

TL;DR

It is shown that highly selective neurons can form already in the early epochs of training and in the early layers of a convolutional spiking network, which is the first time that effective AM has been applied to SNNs.

Abstract

Elements of neural networks, both biological and artificial, can be described by their selectivity for specific cognitive features. Understanding these features is important for understanding the inner workings of neural networks. For a living system, such as a neuron, whose response to a stimulus is unknown and not differentiable, the only way to reveal these features is through a feedback loop that exposes it to a large set of different stimuli. The properties of these stimuli should be varied iteratively in order to maximize the neuronal response. To utilize this feedback loop for a biological neural network, it is important to run it quickly and efficiently in order to reach the stimuli that maximizes certain neurons' activation with the least number of iterations possible. Here we present a framework with an efficient design for such a loop. We successfully tested it on an artificial spiking neural network (SNN), which is a model that simulates the asynchronous spiking activity of neurons in living brains. Our optimization method for activation maximization is based on the low-rank Tensor Train decomposition of the discrete activation function. The optimization space is the latent parameter space of images generated by SN-GAN or VQ-VAE generative models. To our knowledge, this is the first time that effective AM has been applied to SNNs. We track changes in the optimal stimuli for artificial neurons during training and show that highly selective neurons can form already in the early epochs of training and in the early layers of a convolutional spiking network. This formation of refined optimal stimuli is associated with an increase in classification accuracy. Some neurons, especially in the deeper layers, may gradually change the concepts they are selective for during learning, potentially explaining their importance for model performance.
Paper Structure (27 sections, 5 equations, 18 figures)

This paper contains 27 sections, 5 equations, 18 figures.

Table of Contents

  1. Introduction
  2. Paper outline
  3. Background
  4. Activation Maximization in artificial and living neural networks
  5. Spiking Neural Networks (SNNs)
  6. Methods
  7. Tensor Train (TT) decomposition and TT-based optimization techniques
  8. Generative modeling
  9. SNN architectures
  10. Optimization algorithms
  11. MANGO framework
  12. Results
  13. Performance of Tensor Train-based optimization methods
  14. Emergence and dynamics of neuronal specializations
  15. Complexity and diversity of specializations
  16. ...and 12 more sections

Figures (18)

  • Figure 1: Spiking neuron models (image from https://snntorch.readthedocs.io/en/latest/tutorials/index.htmlSNN-Torch-paper)
  • Figure 2: Schematic of the PROTES method.
  • Figure 3: Schematic of the MANGO framework.
  • Figure 4: Activation of one selected neuron (unit 0, first LIF spiking layer, spiking ResNet18) depending on the number of requests to the optimizer. Left: full optimization history, Right: inset for optimization budget from 1000 to 12000.
  • Figure 5: "Pink horse" neuron 52 from LIF layer 1.1 of spiking ResNet18. Images were generated using Tensor Train based methods. Numbers show activations of the target neuron on MEI
  • ...and 13 more figures