TC-LIF: A Two-Compartment Spiking Neuron Model for Long-Term Sequential Modelling

TL;DR

The proposed novel biologically inspired Two-Compartment Leaky Integrate-and-Fire spiking neuron model, dubbed TC-LIF, incorporates carefully designed somatic and dendritic compartments that are tailored to facilitate learning long-term temporal dependencies.

Abstract

The identification of sensory cues associated with potential opportunities and dangers is frequently complicated by unrelated events that separate useful cues by long delays. As a result, it remains a challenging task for state-of-the-art spiking neural networks (SNNs) to establish long-term temporal dependency between distant cues. To address this challenge, we propose a novel biologically inspired Two-Compartment Leaky Integrate-and-Fire spiking neuron model, dubbed TC-LIF. The proposed model incorporates carefully designed somatic and dendritic compartments that are tailored to facilitate learning long-term temporal dependencies. Furthermore, a theoretical analysis is provided to validate the effectiveness of TC-LIF in propagating error gradients over an extended temporal duration. Our experimental results, on a diverse range of temporal classification tasks, demonstrate superior temporal classification capability, rapid training convergence, and high energy efficiency of the proposed TC-LIF model. Therefore, this work opens up a myriad of opportunities for solving challenging temporal processing tasks on emerging neuromorphic computing systems. Our code is publicly available at https://github.com/ZhangShimin1/TC-LIF.

Figures (5)

• Figure 1: Illustration of (a) the structure of a two-compartment Pinsky-Rinzel pyramidal neuron, and the internal operations of (b) LIF model as well as the proposed (c) TC-LIF model. Note that the proposed TC-LIF model in (c) differentiates dendirtic and somatic compartments by highlighting the regions in blue and red, respectively. In contrast, the LIF model in (b) solely incorporates the dynamics of the somatic compartment, without accounting for the dendritic compartment.
• Figure 2: Study the impact of $\beta_1$ and $\beta_2$ initialization on the test accuracy of S-MNIST and PS-MNIST datasets. Note that on both S-MNIST and PS-MNIST datasets, the models initialized in the first and third quadrants face severe exploding and vanishing gradient problems, respectively. As a result, they are unable to learn any meaningful information and are thus stuck at an accuracy of 11.35%. The green dots refer to models that can converge to 100% training accuracy.
• Figure 3: Illustration of the gradient evolution across time on the S-MNIST dataset. Note that the gradient has been calculated on a random batch of $256$ samples with a three-layer feedforward SNN (64-256-256).
• Figure 4: Comparison of the learning curves of TC-LIF and other single-compartment spiking neurons with (a) feedforward and (b) recurrent network architectures. Note that the mean and standard deviations across four runs are reported.
• Figure 5: Comparison of the loss landscape of (a, c) LIF and (b, d) TC-LIF neuron models in terms of 3D surface and 2D contour plots.