SNNLP: Energy-Efficient Natural Language Processing Using Spiking Neural Networks

TL;DR

This work tackles the energy-inefficiency of NLP models by leveraging Spiking Neural Networks (SNNs) and addressing the challenge of encoding text as spike trains. It systematically compares binarized embeddings and rate-coded spike encodings, and introduces a deterministic rate-coding scheme that uses a Self-Accumulate-and-Fire (SAF) neuron variant to improve fidelity without sacrificing spiking efficiency. The results show that deterministic rate-coding improves sentiment and emotion classification performance by about 13% over Poisson rate-coding, narrowing the gap to standard ANNs, while delivering substantial energy savings—roughly 32x during inference and 60x during training—for SNNs. The work also demonstrates that by tuning the inference window, latency can be reduced dramatically (3.7x–7x) with modest accuracy tradeoffs, highlighting the practical potential of neuromorphic NLP for edge devices.

Abstract

As spiking neural networks receive more attention, we look toward applications of this computing paradigm in fields other than computer vision and signal processing. One major field, underexplored in the neuromorphic setting, is Natural Language Processing (NLP), where most state-of-the-art solutions still heavily rely on resource-consuming and power-hungry traditional deep learning architectures. Therefore, it is compelling to design NLP models for neuromorphic architectures due to their low energy requirements, with the additional benefit of a more human-brain-like operating model for processing information. However, one of the biggest issues with bringing NLP to the neuromorphic setting is in properly encoding text into a spike train so that it can be seamlessly handled by both current and future SNN architectures. In this paper, we compare various methods of encoding text as spikes and assess each method's performance in an associated SNN on a downstream NLP task, namely, sentiment analysis. Furthermore, we go on to propose a new method of encoding text as spikes that outperforms a widely-used rate-coding technique, Poisson rate-coding, by around 13\% on our benchmark NLP tasks. Subsequently, we demonstrate the energy efficiency of SNNs implemented in hardware for the sentiment analysis task compared to traditional deep neural networks, observing an energy efficiency increase of more than 32x during inference and 60x during training while incurring the expected energy-performance tradeoff.

Figures (8)

• Figure 1: Overall encoding/training pipeline for both SNN and ANN. Text gets encoded into a continuous representation, converted into a spike train (for SNN usage), and sent to the downstream neural network for training/testing. During training/testing, operation counts are recorded so we can calculate appropriate energy numbers for comparison.
• Figure 2: Illustrated "Self-Accumulate-and-Fire" (SAF) neuron. Takes in a floating-point value and self-accumulates it over time, spiking similarly to a standard LIF neuron.
• Figure 3: All model training curves. The top two graphs depict curves for sentence-level methods and the bottom two graphs show word-level methods. The left-side graphs show models trained for the IMDB 2-class task, where the right-side graphs show models trained for the 6-class emotion classification task.
• Figure 4: SNN vs ANN number of arithmetic operations of Sentiment Classification during inference (a) & training (c) and Emotion Classification during inference (b) & training (d).
• Figure 5: Improvement of SNN in computation energy over ANN. Sentiment Classification during inference (a) and training (c). Emotion Classification during inference (b) and training (d).