Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Equivalence of approximation by networks of single- and multi-spike neurons

Dominik Dold, Philipp Christian Petersen

Abstract

In a spiking neural network, is it enough for each neuron to spike at most once? In recent work, approximation bounds for spiking neural networks have been derived, quantifying how well they can fit target functions. However, these results are only valid for neurons that spike at most once, which is commonly thought to be a strong limitation. Here, we show that the opposite is true for a large class of spiking neuron models, including the commonly used leaky integrate-and-fire model with subtractive reset: for every approximation bound that is valid for a set of multi-spike neural networks, there is an equivalent set of single-spike neural networks with only linearly more neurons (in the maximum number of spikes) for which the bound holds. The same is true for the reverse direction too, showing that regarding their approximation capabilities in general machine learning tasks, single-spike and multi-spike neural networks are equivalent. Consequently, many approximation results in the literature for single-spike neural networks also hold for the multi-spike case.

Equivalence of approximation by networks of single- and multi-spike neurons

Abstract

In a spiking neural network, is it enough for each neuron to spike at most once? In recent work, approximation bounds for spiking neural networks have been derived, quantifying how well they can fit target functions. However, these results are only valid for neurons that spike at most once, which is commonly thought to be a strong limitation. Here, we show that the opposite is true for a large class of spiking neuron models, including the commonly used leaky integrate-and-fire model with subtractive reset: for every approximation bound that is valid for a set of multi-spike neural networks, there is an equivalent set of single-spike neural networks with only linearly more neurons (in the maximum number of spikes) for which the bound holds. The same is true for the reverse direction too, showing that regarding their approximation capabilities in general machine learning tasks, single-spike and multi-spike neural networks are equivalent. Consequently, many approximation results in the literature for single-spike neural networks also hold for the multi-spike case.
Paper Structure (13 sections, 4 theorems, 7 equations, 1 figure)

This paper contains 13 sections, 4 theorems, 7 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Background
  3. Learning task
  4. Spiking neuron models
  5. Network architectures
  6. Bounded number of spikes
  7. Results
  8. Proofs
  9. Proof of the first part.
  10. Proof of the second part.
  11. Proof.
  12. Corollaries.
  13. Limitations

Key Result

Theorem 1

Let $d_\text{in},d_\text{out},N_\text{s} \in \mathbb{N}$, $f: S_{\infty}^{d_\text{in}} \to S_{N_\text{s}}^{d_\text{out}}$ be time-causal, and $F = D \circ f$ with decoder $D$. Furthermore, set $\Omega \subseteq S_{\infty}^{d_\text{in}}$. Then, the following hold for each $p \in (0, \infty]$ and eac

Figures (1)

  • Figure 1: Illustration of how to construct equivalent single-and multi-spike SNNs, shown here for multi-spike neurons that spike at most six times in a given finite time interval. A. A multi-spike neuron can be replaced by a population of single-spike neurons that produce an equivalent spike output. B. The $k^\text{th}$ single-spike neuron spikes at the $k^\text{th}$ spike time of the multi-spike neuron. This is achieved by setting the threshold of each neuron accordingly (alternatively, the reset potential could be adjusted). C. A single-spike neuron can also be replaced by a population of multi-spike neurons. D. Given any input, the first neuron produces a spike train. We add additional neurons to the population that gain identical input, but have increasing thresholds. For instance, the second neuron only spikes at every second spike time of the first neuron. By choosing synaptic weights accordingly, all but the first spike of the first neuron are cancelled out, producing an effective spike train with only one spike.

Theorems & Definitions (4)

  • Theorem 1: Transference principle
  • Corollary 1: Lower bounds
  • Corollary 2: Encoders
  • Lemma 1