Mitigating Communication Costs in Neural Networks: The Role of Dendritic Nonlinearity

TL;DR

The paper investigates whether nonlinear dendritic processing can reduce communication costs in artificial neural networks without substantially harming learning capacity. It introduces a dendritic neuron model with $K$ branches and compares it to point neurons across dense and sparse regimes, maintaining comparable compute by setting $\hat{D}=D/\sqrt{K}$ and using budget ratio $\Psi$. The key finding is that dendritic nonlinearities provide limited gains in learning capacity but substantially lower inter-neuronal communication and memory access, with empirical scaling $\hat{C}_E \propto K^{-0.51}$ under fixed budgets. These results inform the design of energy-efficient neural accelerators and memory systems for training and inference.

Abstract

Our understanding of biological neuronal networks has profoundly influenced the development of artificial neural networks (ANNs). However, neurons utilized in ANNs differ considerably from their biological counterparts, primarily due to the absence of complex dendritic trees with local nonlinearities. Early studies have suggested that dendritic nonlinearities could substantially improve the learning capabilities of neural network models. In this study, we systematically examined the role of nonlinear dendrites within neural networks. Utilizing machine-learning methodologies, we assessed how dendritic nonlinearities influence neural network performance. Our findings demonstrate that dendritic nonlinearities do not substantially affect learning capacity; rather, their primary benefit lies in enabling network capacity expansion while minimizing communication costs through effective localized feature aggregation. This research provides critical insights with significant implications for designing future neural network accelerators aimed at reducing communication overhead during neural network training and inference.

Figures (14)

• Figure 1: (A, B, C) Illustration of three representative neurons showcasing distinct dendritic structures from left to right: A chicken bipolar neuron wang2012vivo, a human hippocampal pyramidal neuron benavides2020differential, and a ferret neocortical pyramidal neuron adusei2021morphological. All neuronal morphologies are from the Neuromorpho.org database ascoli2007neuromorpho. (D) Portrays a point neuron, as characterized by Equation \ref{['equ:neuron']}. (E) Illustrates a dendritic neuron with $4$ dendritic branches as detailed by Equations \ref{['equ:den']} and \ref{['equ:dendrite']}.
• Figure 2: Comparison of neural network layers using point neurons (Top) and dendritic neurons (Bottom). Only two layers from each model are depicted. The point neuron model has $D=8$ channels, whereas the dendritic neuron model features neurons with $K=4$ dendritic branches each, leading to an effective $\hat{D} = \frac{D}{\sqrt{4}} = 4$ channels. This ensures that both models have comparable parametric and computational complexities. Note: Tensor dimensions are symbolized by a mesh of patches; however, patch sizes do not reflect actual scale. Bias terms have been excluded for simplicity.
• Figure 3: Comparison of ResNet-18-style models using point vs. dendritic neurons on ImageNet. Left (A-C): Experiment on dense models (5 trials, std dev shown). The red dot (①) marks the baseline ResNet-18 with standard point neurons (K=1). The x-axis indicates the number of dendrites per neuron (K). Three complexity levels are evaluated: Standard complexity (light blue dashed curves), models with the same complexity as baseline ResNet-18. 4x complexity (solid magenta curves). 16x complexity (brown dashed curves). Within each curve, models share the same total parametric budget but differ in $K$ (number of dendrites per neuron). For example, model (②) is configured with a $\Psi$ ratio of 0.5 to match the complexity of the baseline. Model (③), with $\Psi=1$ and $K=4$, has $4\times$ the complexity of the baseline model (①). (A) Training set accuracy. (B) Test set accuracy. (C)$\Psi$ ratio relative to the baseline ResNet-18. Right (D-F): Same layout and analysis, but for sparse models (3 trials; standard deviation shown).
• Figure 4: Illustration of the three communication cost metrics ($C_A$, $D$, and $C_E$) for (A) Biological Neural Networks and (B) ANNs. In this context, $C_A$ represents the communication cost associated with aggregating synaptic inputs; $D$ denotes the inter-layer communication cost, and $C_E$ signifies the expense related to signal propagation to each synapse (weight). We measure $C_A$ and $C_E$ by the total path length over which the signal traverses. For the ANN models, we assume model inference is performed on a mesh of processing elements (PEs). Each blue dot represent one PE unit in the mesh. It's important to note that the division into these three metrics is not intended to be exact.
• Figure 5: (A, B) Estimation of signal propagation costs $C_E$ for a biological network layer with a varying number of dendrites per neuron ($K$) and a baseline network of dimension $D=1024$. Post-synaptic targets sampled from (A) a unit square. (B) from a unit cube. In each panel, the curve and its corresponding equation are fitted to the data points. (C,D) Estimation of signal propagation costs for a ANN layer. (C) Topographic representation of the ratio $\eta=({\hat{C}_A+\hat{C}_E})/({C_A+C_E})$: The visualization highlights the influence of the variations in $D$ and $K$ on the $\eta$. (D) demonstrates the variations in $\hat{C}_E$ as a function of $\sqrt{K}$, and levels of connection sparsity. The axes are depicted on a logarithmic scale. When $K=1$, the models are based on point neurons. For this experiment, a $D$ value of 256 was utilized. The slope is obtained from fitting a line to the logarithm of $C_E$ against the logarithm of $\sqrt{K}$.

Theorems & Definitions (4)

• Theorem 1
• Lemma 1
• proof
• proof : Proof of Theorem \ref{['thm:1']}: