Skewed Neuronal Heterogeneity Enhances Efficiency On Various Computing Systems

TL;DR

This work investigates whether intrinsic, task-agnostic heterogeneity in neuronal membrane time constants can lift temporal computation without task-specific tuning. Using rate-based and spiking networks within a reservoir-computing framework, the authors show that skewed time-constant heterogeneity enhances accuracy, robustness, and energy efficiency across hundreds of temporal tasks and across in silico and neuromorphic platforms. A comprehensive, low-bias task benchmark demonstrates broad improvements not tied to a particular stimulus, while analyses reveal that performance tracks with improved task-state alignment rather than merely higher state dimensionality. The findings imply that biological heterogeneity can serve as a powerful inductive bias for designing efficient artificial systems and neuromorphic devices that exploit device-to-device variability rather than suppress it.

Abstract

Heterogeneity is a ubiquitous property of many biological systems and has profound implications for computation. While it is conceivable to optimize neuronal and synaptic heterogeneity for a specific task, such top-down optimization is biologically implausible, prone to catastrophic forgetting, and both data- and energy-intensive. In contrast, biological organisms, with remarkable capacity to perform numerous tasks with minimal metabolic cost, exhibit a heterogeneity that is inherent, stable during adulthood, and task-unspecific. Inspired by this intrinsic form of heterogeneity, we investigate the utility of variations in neuronal time constants for solving hundreds of distinct temporal tasks of varying complexity. Our results show that intrinsic heterogeneity significantly enhances performance and robustness in an implementation-independent manner, indicating its usefulness for both (rate-based) machine learning and (spike-coded) neuromorphic applications. Importantly, only skewed heterogeneity profiles-reminiscent of those found in biology-produce such performance gains. We further demonstrate that this computational advantage eliminates the need for large networks, allowing comparable performance with substantially lower operational, metabolic, and energetic costs, respectively in silico, in vivo, and on neuromorphic hardware. Finally, we discuss the implications of intrinsic (rather than task-induced) heterogeneity for the design of efficient artificial systems, particularly novel neuromorphic devices that exhibit similar device-to-device variability.

Figures (26)

• Figure 1: Method summary. (A) Schematic of heterogeneous networks. (Left) Membrane time constants of neurons are drawn from a log‑normal distribution with a fixed mean; the variance controls the level of heterogeneity. (Right) Apart from the neuronal time constants (values are indicated by different radii), the networks have identical connectivity and inputs. (B) A multi-dimensional chaotic time series $\bm u$, mimicking partially predictable multi-modal sensory stimuli, drives the network dynamics. In addition, each neuron receives white noise with $10\%$ of the stimulus variance (not shown). (C) Schematic of the reservoir computing paradigm. The network’s state $\bm X(t)$ is used to reconstruct a set of target functions $y_{\theta}(t)$ via the optimal task‑specific linear readout $\bm \beta_{\theta}$. (D) Tasks belong to a family $\mathcal{F}_{\theta}$, parameterized by $\theta = (\Delta, d, k)$, which captures working‑memory‑like computations such as memory recall, forecasting, and nonlinear processing by transforming the stimulus $\bm u(t)$. Traces show some exemplary target functions. (E) Network outputs, $\hat{y}_{\theta}(t)=\bm \beta_{\theta}^{\top}\!\cdot\!\bm X(t)$, for an exemplary task. Colors indicate different heterogeneity levels as in (A). The ground‑truth $y_{\theta}(t)$ (thick gray line) is shown for comparison. (F) Performance associated with the traces shown in (E). Performance of each network is measured by the coefficient of determination, a scalar score that quantifies the normalized mismatch between the ground truth and the output over an unobserved test interval. A value of 1 indicates a perfect match, while 0 corresponds to chance‑level performance. Colors as in (E).
• Figure 2: Intrinsically heterogeneous networks process chaotic time series more accurately. The task family $y_{\theta}=u_k(t+\Delta)^d$ was used to generate 882 tasks with parameters drawn from the ranges $d\in\{1,\dots,6\}$, $\Delta\in[-2,2]$, and $k\in\{1,2,3\}$. Performance of all networks was evaluated on each of these tasks. (A) Distribution of cosine similarities between all task pairs; most pairs are orthogonal and therefore temporally independent. (B) Task complexity is defined as the cosine dissimilarity between the target function $u_k(t+\Delta)^d$ and its source input $u_k(t)$. This definition assigns a minimal complexity of 0 to the identity (or pure‑relay) task, which requires neither memory ($\Delta=0$) nor nonlinear processing ($d=1$). Tasks in $\mathcal{F}_{\theta}$ therefore span a broad spectrum of complexities. (C) Performance profiles for a subset of parameters corresponding to quadratic ($d=2$) recall ($\Delta<0$) and forecasting ($\Delta>0$) of the first ($k=1$) input component. The top and bottom panels show the results for spiking (LIF) and rate‑based (LI) implementations, respectively. Shaded area show 10-fold enlarged standard deviation estimated over three independent trials. (D) Direct score comparison between homogeneous (x‑axis) and heterogeneous (y‑axis) networks for each task (each dot). Colors indicate the degree of heterogeneity. The diagonal dotted line marks equal performance, while the vertical and horizontal dashed lines denote the chance‑level score. The right‑hand panel corresponds to the spiking implementation and the left‑hand panel to the rate‑based implementation. Panels (C) and (D) share the same color code.
• Figure 3: Neuronal heterogeneity renders networks robust.(A)(Top) Schematic of the network and its inputs. Purple and green traces represent the deterministic input and stochastic noise that are projected onto the network at its center. (Bottom) Tasks are divided into three complexity tiers (indicated by the gradient), and performance is averaged within each tier for clarity. (B–E)(Top) Schematic of the varied hyperparameters; see panel (A) for reference. (Bottom) Impact of varying network size (B), noise intensity (C), feedforward gain (D), and recurrent gain (E) on the performance of rate‑based RNNs. All panels use the same color code. The plateaus in panel (D) arise from saturation of neuronal activity. The uniform decline of the score at large recurrent‑gain values in panel (E) reflects a transition to chaos. Analogous results for spiking networks are shown in Fig. \ref{['fig:supp_robustness_lif']}. In all panels dots represents the simulated hyperparameters and the shaded area indicate 95% confidence interval, calculated over tasks belonging to the corresponding complexity tier.
• Figure 4: Heterogeneity lowers the cost at each performance level. At every performance bin, the network with minimal energy consumption ($E^{*}$) is the most efficient. In all systems, heterogeneity substantially reduces the total minimal cost (note the logarithmic scale of the y‑axis). This cost saving primarily results from using equally performant networks that are much smaller. The definition of cost for each hardware system reflects its specific characteristics. We used (A) the total number of floating‑point operations (FLOPs) and the memory footprint for rate‑based networks, and (B) the total expenditure of adenosine triphosphate (ATP) molecules in biological spiking networks. For neuromorphic systems, (C) Loihi 2 and (D) SpiNNaker 2, the total energy consumption (in joules) was estimated based on direct measurements of their power usage.
• Figure 5: The heterogeneity profile substantially affects generalization.(A) At any given value of $\mathop{\mathrm{\mathbb{E}}}\nolimits[\tau]$, heterogeneity improves performance. By spreading neuronal time constants over a broader interval, heterogeneity increases the likelihood of covering the regions required by the tasks. (B) The performance depends on the heterogeneity profile. Skewed distributions (log‑normal and gamma) result in higher performance levels. The empirical mean and variance were identical for all profiles. The shaded area in (A) and (B) correspond to indicate 95% confidence interval across all tasks.