Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Scalable, Causal, and Energy Efficient Framework for Neural Decoding with Spiking Neural Networks

Georgios Mentzelopoulos, Ioannis Asmanis, Konrad P. Kording, Eva L. Dyer, Kostas Daniilidis, Flavia Vitale

TL;DR

Spikachu presents a scalable, causal, and energy-efficient neural decoding framework for BCIs built on spiking neural networks. By harmonizing cross-session/subject data into a shared latent space and deploying multi-temporal spiking modules with spiking self-attention, it achieves competitive decoding performance while dramatically reducing energy consumption. The authors show that pretraining on large, heterogeneous neural datasets (Spikachu-mp) yields robust representations that transfer to new sessions and subjects with faster convergence and additional energy savings. These results highlight the practicality of energy-conscious, online-compatible neural decoding for edge-enabled BCIs and motivate further validation on neuromorphic hardware for real-world clinical deployment.

Abstract

Brain-computer interfaces (BCIs) promise to enable vital functions, such as speech and prosthetic control, for individuals with neuromotor impairments. Central to their success are neural decoders, models that map neural activity to intended behavior. Current learning-based decoding approaches fall into two classes: simple, causal models that lack generalization, or complex, non-causal models that generalize and scale offline but struggle in real-time settings. Both face a common challenge, their reliance on power-hungry artificial neural network backbones, which makes integration into real-world, resource-limited systems difficult. Spiking neural networks (SNNs) offer a promising alternative. Because they operate causally these models are suitable for real-time use, and their low energy demands make them ideal for battery-constrained environments. To this end, we introduce Spikachu: a scalable, causal, and energy-efficient neural decoding framework based on SNNs. Our approach processes binned spikes directly by projecting them into a shared latent space, where spiking modules, adapted to the timing of the input, extract relevant features; these latent representations are then integrated and decoded to generate behavioral predictions. We evaluate our approach on 113 recording sessions from 6 non-human primates, totaling 43 hours of recordings. Our method outperforms causal baselines when trained on single sessions using between 2.26 and 418.81 times less energy. Furthermore, we demonstrate that scaling up training to multiple sessions and subjects improves performance and enables few-shot transfer to unseen sessions, subjects, and tasks. Overall, Spikachu introduces a scalable, online-compatible neural decoding framework based on SNNs, whose performance is competitive relative to state-of-the-art models while consuming orders of magnitude less energy.

A Scalable, Causal, and Energy Efficient Framework for Neural Decoding with Spiking Neural Networks

TL;DR

Spikachu presents a scalable, causal, and energy-efficient neural decoding framework for BCIs built on spiking neural networks. By harmonizing cross-session/subject data into a shared latent space and deploying multi-temporal spiking modules with spiking self-attention, it achieves competitive decoding performance while dramatically reducing energy consumption. The authors show that pretraining on large, heterogeneous neural datasets (Spikachu-mp) yields robust representations that transfer to new sessions and subjects with faster convergence and additional energy savings. These results highlight the practicality of energy-conscious, online-compatible neural decoding for edge-enabled BCIs and motivate further validation on neuromorphic hardware for real-world clinical deployment.

Abstract

Brain-computer interfaces (BCIs) promise to enable vital functions, such as speech and prosthetic control, for individuals with neuromotor impairments. Central to their success are neural decoders, models that map neural activity to intended behavior. Current learning-based decoding approaches fall into two classes: simple, causal models that lack generalization, or complex, non-causal models that generalize and scale offline but struggle in real-time settings. Both face a common challenge, their reliance on power-hungry artificial neural network backbones, which makes integration into real-world, resource-limited systems difficult. Spiking neural networks (SNNs) offer a promising alternative. Because they operate causally these models are suitable for real-time use, and their low energy demands make them ideal for battery-constrained environments. To this end, we introduce Spikachu: a scalable, causal, and energy-efficient neural decoding framework based on SNNs. Our approach processes binned spikes directly by projecting them into a shared latent space, where spiking modules, adapted to the timing of the input, extract relevant features; these latent representations are then integrated and decoded to generate behavioral predictions. We evaluate our approach on 113 recording sessions from 6 non-human primates, totaling 43 hours of recordings. Our method outperforms causal baselines when trained on single sessions using between 2.26 and 418.81 times less energy. Furthermore, we demonstrate that scaling up training to multiple sessions and subjects improves performance and enables few-shot transfer to unseen sessions, subjects, and tasks. Overall, Spikachu introduces a scalable, online-compatible neural decoding framework based on SNNs, whose performance is competitive relative to state-of-the-art models while consuming orders of magnitude less energy.
Paper Structure (90 sections, 35 equations, 12 figures, 6 tables)

This paper contains 90 sections, 35 equations, 12 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural decoding.
  4. Spiking neural networks.
  5. Methodology
  6. Harmonizing the neural activity across sessions and subjects
  7. Tokenization.
  8. Projecting units to a shared latent space.
  9. Obtaining virtual units.
  10. Efficient and stateful processing of the latents using spiking neural networks
  11. Decomposing the neural activity across temporal streams.
  12. Capturing long-range temporal dependencies.
  13. Compressing the latents to a compact spatiotemporal representation.
  14. Smoothing predictions using multiple timescales.
  15. Tracking continuous variables through the membrane potential of spiking neurons.
  16. ...and 75 more sections

Figures (12)

  • Figure 1: Overview of the Spikachu framework.
  • Figure 2: Model performance on single sessions. (A) Schematic of CO and RT tasks (reproduced from Azabou2023, with permission). (B) Examples of true and predicted behavior (x, y velocities) for the CO (top 4 subplots) and RT (bottom 4 subplots) tasks. (C) Mean ($\pm$ SEM) decoding performance and energy consumption for Spikachu and baselines on sessions from monkeys C, J, and M from the Perich2018 dataset.
  • Figure 3: Head-to-head comparison between Spikachu-mp + finetuning vs Spikachu trained on single-sessions.(A) Decoding Performance, (B) Energy consumption per inference. Marker size encodes the number of trials available for each session.
  • Figure 4: Performance comparison between single-session models trained from scratch vs single-session model transferred from Spikachu-mp.(A). Decoding Performance, (B) Energy consumption per inference, (C, D) Learning dynamics for the CO and RT tasks.
  • Figure 5: Benefits of scaling up the training dataset size.(A, B) Decoding performance of finetuned and transferred models (performance of scratch-trained single-session models overlayed in gray) as a function of the number of sessions used for model pretraining for the (A) CO, and (B) RT tasks. Panels (C, D) show the energy consumption per inference for finetuned and transferred models (performance of scratch-trained single-session models overlayed in gray) as a function of the number of sessions used for model pretraining for the (C) CO, and (D) RT tasks.
  • ...and 7 more figures