Causal pieces: analysing and improving spiking neural networks piece by piece

TL;DR

This paper introduces the concept of causal pieces in spiking neural networks (SNNs), defining regions where the causal path remains fixed and showing that output spike times are locally Lipschitz within these regions; the number of causal pieces serves as a measure of expressiveness and can predict training success. By analyzing a non-leaky integrate-and-fire (nLIF) model with exponential synapses, the authors derive causal sets and paths, prove Lipschitz continuity within pieces, and develop counting methods, including a data-informed practical count and a random-walk-based upper bound. Empirically, they demonstrate that higher initial piece counts correlate with better training outcomes, that deeper networks increase piece counts (with logistic growth patterns), and that networks with exclusively positive weights can still achieve competitive performance when piece structure is favorable. These results offer a principled tool for initializing, designing, and evaluating SNNs, with potential implications for energy efficiency and comparisons to artificial neural networks (ANNs).

Abstract

We introduce a novel concept for spiking neural networks (SNNs) derived from the idea of "linear pieces" used to analyse the expressiveness and trainability of artificial neural networks (ANNs). We prove that the input domain of SNNs decomposes into distinct causal regions where its output spike times are locally Lipschitz continuous with respect to the input spike times and network parameters. The number of such regions - which we call "causal pieces" - is a measure of the approximation capabilities of SNNs. In particular, we demonstrate in simulation that parameter initialisations which yield a high number of causal pieces on the training set strongly correlate with SNN training success. Moreover, we find that feedforward SNNs with purely positive weights exhibit a surprisingly high number of causal pieces, allowing them to achieve competitive performance levels on benchmark tasks. We believe that causal pieces are not only a powerful and principled tool for improving SNNs, but might also open up new ways of comparing SNNs and ANNs in the future.

Figures (6)

• Figure 1: Causal sets and causal pieces. (A) Causal sets contain all constituents that caused an output spike. (top) A single output neuron (orange) receiving input from three neurons. Only those input neurons that spike before the output neuron (i.e., before dotted line, dark gray) are part of the causal set. (bottom) In deep networks, this corresponds to paths through the network, here shown for a single output neuron (left, orange), or the whole output layer (right, orange and blue). (B) Illustration of causal pieces of a single neuron. The output spike time of the neuron when following the x-axis is shown at the bottom. (C) Causal pieces of the output neuron for two networks with different depth.
• Figure 2: Estimating the number of causal pieces. (A) The probabilities $p^q_k$ are obtained by counting how many trajectories (cumulative sum of weights) are above the threshold at step $k$. The top panel shows the corresponding values of $p^q_k$, where $k$ is the number of weights. (B) Estimated number of pieces for weights sampled from normal distribution with different mean (y-axis) and standard deviation (x-axis). Colours are shown in log-scale. (C)$p^q_k$ for two points in (B), denoted by markers.
• Figure 3: Network initialization strongly affects training success. (A) The logarithm of the number of pieces (here: of the output layer) at network initialization strongly correlates with performance after training ($r = 0.94$). The correlation between pieces and accuracy is $r = 0.77$. (B) Same as (A), but with the number of pieces after training. For pieces vs. accuracy, we find $r = 0.81$. (C) Median causal set size depending on the number of causal pieces before (blue) and after (red) training. (D) Number of pieces before and after training. The diagonal indicates no change in pieces. (E) Illustration of the Yin Yang dataset with three classes: the two halves and the dots. (F) Causal pieces (each piece is indicated by a different colour) of a single output neuron for a bad initialization, evaluated using only training samples. (G) Same as (F), but for one of the best initializations.
• Figure 4: Width-and depth dependence of causal pieces. (A)$p^q_k$ of the optimized (top, dot) normal, and (bottom, square) uniform initialization. (B) Number of pieces for shallow and deep networks. The maximum number, which is the number of input samples used to evaluate the number of causal pieces, is shown as a dash-dotted line. (C) Number of pieces per layer in a single network, before and after training. (D) Increase in the total number of pieces for deep and shallow networks. Markers denote results that belong together. We show medians (lines) and quartiles (shaded areas).
• Figure 5: Causal pieces (coloured regions) of one of the output neurons for an nLIF neural network with $[4, 30, 3]$ neurons, using the initializations obtained through evolutionary optimization (\ref{['fig:Deep']} and \ref{['fig:PosSNN']}). Causal pieces are evaluated using a 400 $\times$ 400 grid on the data domain.

Theorems & Definitions (8)

• Definition 1: nLIF
• Definition 2: Causal set
• Definition 3: Causal path
• Definition 4: Causal piece
• Theorem 1: Lipschitz continuous
• Theorem 2: Approximation bound
• Theorem 3: Number of pieces in limit
• Theorem 4