Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neuromorphic Computing: A Theoretical Framework for Time, Space, and Energy Scaling

James B Aimone

Abstract

Neuromorphic computing (NMC) is increasingly viewed as a low-power alternative to conventional von Neumann architectures such as central processing units (CPUs) and graphics processing units (GPUs), however the computational value proposition has been difficult to define precisely. Here, we propose a computational framework for analyzing NMC algorithms and architectures. Using this framework, we demonstrate that NMC can be analyzed as general-purpose and programmable even though it differs considerably from a conventional stored-program architecture. We show that the time and space scaling of idealized NMC has comparable time and footprint tradeoffs that align with that of a theoretically infinite processor conventional system. In contrast, energy scaling for NMC is significantly different than conventional systems, as NMC energy costs are event-driven. Using this framework, we show that while energy in conventional systems is largely determined by the scheduled operations determined by the structural algorithm graph, the energy of neuromorphic systems scales with the activity of the algorithm, that is the activity trace of the algorithm graph. Without making strong assumptions on NMC or conventional costs, we demonstrate which neuromorphic algorithm formulations can exhibit asymptotically improved energy scaling when activity is sparse and decaying over time. We further use these results to identify which broad algorithm families are more or less suitable for NMC approaches.

Neuromorphic Computing: A Theoretical Framework for Time, Space, and Energy Scaling

Abstract

Neuromorphic computing (NMC) is increasingly viewed as a low-power alternative to conventional von Neumann architectures such as central processing units (CPUs) and graphics processing units (GPUs), however the computational value proposition has been difficult to define precisely. Here, we propose a computational framework for analyzing NMC algorithms and architectures. Using this framework, we demonstrate that NMC can be analyzed as general-purpose and programmable even though it differs considerably from a conventional stored-program architecture. We show that the time and space scaling of idealized NMC has comparable time and footprint tradeoffs that align with that of a theoretically infinite processor conventional system. In contrast, energy scaling for NMC is significantly different than conventional systems, as NMC energy costs are event-driven. Using this framework, we show that while energy in conventional systems is largely determined by the scheduled operations determined by the structural algorithm graph, the energy of neuromorphic systems scales with the activity of the algorithm, that is the activity trace of the algorithm graph. Without making strong assumptions on NMC or conventional costs, we demonstrate which neuromorphic algorithm formulations can exhibit asymptotically improved energy scaling when activity is sparse and decaying over time. We further use these results to identify which broad algorithm families are more or less suitable for NMC approaches.

Paper Structure

This paper contains 62 sections, 18 theorems, 114 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Previous Work and Contributions
  3. Defining Neuromorphic Architectures in the Context of Von Neumann
  4. Strengths and weaknesses of Von Neumann
  5. Formal model definitions for neuromorphic computing
  6. Ideal versus realizable neuromorphic architectures and algorithms
  7. Overview of analysis
  8. The Benefits and Costs of Moving to Neuromorphic from Von Neumann
  9. Neuromorphic computing is programmable, but it is not a stored-program architecture
  10. Neuromorphic computing is intrinsically parallel and asynchronous
  11. Neuromorphic processing is data-dependent and often non-deterministic
  12. What neuromorphic is good at
  13. Time and space: neuromorphic operates at depth-limits of extreme parallelism but with associated footprint cost
  14. Energy: neuromorphic costs scale with activity, not work
  15. What does $\Delta G$ mean?
  16. ...and 47 more sections

Key Result

Lemma M2.2

Let $G_N$ be a neuromorphic algorithm graph executed for a horizon $T$ under a fixed execution semantics, and let $G_N^{(T)}$ denote its time-unrolled computational graph (Definition def:unroll. Suppose an idealized neuromorphic substrate in which (i) each vertex update in $G_N^{(T)}$ has unit time This statement is with respect to the specific instantiated neuromorphic graph $G_N$ (via its unrol

Figures (5)

  • Figure 1: (A) NMC hardware is GPU-like in generality, but shows advantageous capabilities in a different set of tasks. Listed are example algorithm features that can lend advantages (which this framework will explore); though note that algorithms can often be reformulated to fit an architecture. (B) An example algorithm graph that has six sets of instructions that require $T_i$ serial operations each. (C) A serial Von Neumann system stores both instructions and data in a single memory structure (RAM), iterating through the algorithm (CPU). (D) An ideal NMC processor spatially distributes the entire algorithm across hardware with a spatial cost proportional to the serial depth, $T_i$, for each operation.
  • Figure 2: Illustration of model framework. (A) Biological and artificial neurons generally have the same components: synapses which exhibit synaptic dynamics and learning; dendrites and the soma which exhibit internal state dynamics, and spike generation at the soma / axon boundary. (B) The framework treats each of these dynamics as a graph of composable computational elements within each neuron. All neurons own their input synaptic dynamics (both pre- and post-synaptic components; $n_S$), all dendritic and somatic dynamics ($n_X$), and spike generation ($n_Y$). (C) The set of all neuron parts, referred to as the neurons, $N$, represents the totality of the dynamics of a neural algorithm. (D) The set of all connections between neurons, referred to here as the synapses, $S$, represents the interconnect between neurons. A neural algorithm is described as an algorithm graph of these neurons (vertices) and synapses (edges), $G_N=(N,S)$
  • Figure 3: Notional landscape of algorithm classes against framework's structural and trace algorithm metrics. Left: Structural metrics, including within-time homogeneity ($H_T$) and across-time structural reuse ($R_T$), can be used to differentiate SIMD- and NMC-friendly algorithms. Right: Trace metrics, including baseline activity ($\Phi_S$) and activity variability ($CV_S$), provide a separate axis for assessing NMC friendliness. Note that the placement of these different algorithm classes is notional; as the placement can vary considerably with input data and the specific application.
  • Figure M1.4: Illustration of structural model, $G_N=(N,S)$ of neural graphs used in framework. For a neural algorithm $G_N=(N,S)$, $\mathcal{N}$ (top left) represents the set of all neurons, $S$ (bottom left) represents the set of all synaptic connections, and $N$ (right) represents the set of the neuron parts which are responsible for all of the computation in the neural algorithm, including $N_S$ (all synaptic computation), $N_X$ (all internal neuronal dynamics), and $N_Y$ (all spiking dynamics), as well as $N_E$ which describes internal neuronal communication.
  • Figure M2.5: Illustration of trace model, $\Delta G_N=(\Delta N, \Delta S)$ of neural graphs used in framework. For a neural algorithm $G_N=(N,S)$, $\mathcal{\Delta N}$ (top left) represents the set of all neurons whose state has changed at a point in time, $\Delta S$ (bottom left) represents the set of all synaptic connections whose state has changed, and $\Delta N$ (right) represents the set of the neuron parts which consist of all of the computation elements that have changed in the neural algorithm, including $\Delta N_S$ (all synaptic computation that has changed), $\Delta N_X$ (all internal neuronal dynamics that has changed), and $\Delta N_Y$ (all spiking dynamics that has changed), as well as $\Delta N_E$ which describes internal neuronal communication that has changed.

Theorems & Definitions (71)

  • Definition M1.1: Neuron
  • Definition M1.2: Synapse
  • Remark M1.3
  • Remark M1.4
  • Definition M1.5: Local stochasticity primitive
  • Definition M1.6: Neuromorphic Computing Architecture
  • Definition M1.7: Neuromorphic Algorithm
  • Definition M1.8: Intermediate Computational Graph
  • Definition M1.9: Relative Complexity Gain
  • Definition M1.10: Neuromorphic Advantage
  • ...and 61 more