Sustainable AI: Mathematical Foundations of Spiking Neural Networks

TL;DR

This paper addresses the sustainability challenge of deep learning by examining spiking neural networks (SNNs) as energy-efficient alternatives to artificial neural networks (ANNs). It builds a formal framework distinguishing continuous versus discrete time and spike-based versus rate-based information, and analyzes two representative models—SRM with TTFS encoding and discretized LIF (DLIF) with rate coding—across expressivity, training, generalization, and energy efficiency. The authors establish universal approximation results for SRM variants, articulate training regimes including surrogate gradients and backpropagation through time, and compare DLIF and SRM TTFS in terms of learning dynamics and complexity, linking these properties to energy considerations. They further discuss generalization bounds within statistical learning theory and lay out an energy-modeling framework that highlights the trade-offs between computation, memory access, and hardware design, arguing for co-design of algorithms and neuromorphic hardware to realize the potential energy benefits of SNNs in practice.

Abstract

Deep learning's success comes with growing energy demands, raising concerns about the long-term sustainability of the field. Spiking neural networks, inspired by biological neurons, offer a promising alternative with potential computational and energy-efficiency gains. This article examines the computational properties of spiking networks through the lens of learning theory, focusing on expressivity, training, and generalization, as well as energy-efficient implementations while comparing them to artificial neural networks. By categorizing spiking models based on time representation and information encoding, we highlight their strengths, challenges, and potential as an alternative computational paradigm.

Figures (3)

• Figure 1: A schematic illustrating the design choices in deriving the considered SNN models and highlighting their application in analyzing the expressivity, training, generalization, and energy-efficiency of SNNs.
• Figure 2: Illustration of different response functions in the SRM model. (a) Biologically realistic response function $\varepsilon_\alpha$ with parameter $\tau_c$ and weights $w$. (b) Simplified response functions abstracted from $\varepsilon_\alpha$, including a piecewise linear function with cutoff at $\delta=0.5$, a step function, and the ReLU activation ($\delta=\infty$). The delay as an additional parameter allows for shifted versions along the time dimension.
• Figure 3: (a) Relative contributions of (distant/local) memory accesses and compute operations to energy consumption in neurons; ANN-opt is optimized towards local memory usage which is more efficient than distant memory leading to the relative increase of the compute costs. (b) Energy efficiency of DLIF SNNs ($T=1$) relative to ANNs as a function of $N_{\text{spikes/syn}}$ using the AlexNet network topology. ANN-base and ANN-opt denote the worst and best case, respectively, i.e., existing ANN accelerators approach the best case. SNNs achieve superior energy efficiency below $N_{\text{spikes/syn}} \approx 0.4$. Both figures are adapted from Dampfhoffer22Energy.

Theorems & Definitions (2)

• Theorem 1
• Theorem 2