LMUFormer: Low Complexity Yet Powerful Spiking Model With Legendre Memory Units

TL;DR

This paper introduces LMUFormer, a low-complexity sequential model that augments Legendre Memory Units with convolutional patch embedding and channel mixers to approximate Transformer-level performance while preserving sequential processing and enabling efficient training. A spiking variant, Spiking LMUFormer, further enhances energy efficiency by leveraging spike-based computation and careful integration with patch embedding and LMU states. Across permuted sequential MNIST, Speech Commands, and Long Range Arena benchmarks, LMUFormer achieves competitive or superior accuracy with dramatically reduced parameters ($\approx$53x fewer) and FLOPs (≈65x fewer) compared to state-of-the-art Transformer baselines, while the Spiking LMUFormer attains SOTA-like results on SNN benchmarks. The results demonstrate the viability of hybrid LMU-based architectures for streaming, edge-friendly sequence learning, combining memory-based temporal processing with lightweight convolutional front-ends and energy-aware spiking dynamics.

Abstract

Transformer models have demonstrated high accuracy in numerous applications but have high complexity and lack sequential processing capability making them ill-suited for many streaming applications at the edge where devices are heavily resource-constrained. Thus motivated, many researchers have proposed reformulating the transformer models as RNN modules which modify the self-attention computation with explicit states. However, these approaches often incur significant performance degradation. The ultimate goal is to develop a model that has the following properties: parallel training, streaming and low-cost inference, and SOTA performance. In this paper, we propose a new direction to achieve this goal. We show how architectural modifications to a recurrent model can help push its performance toward Transformer models while retaining its sequential processing capability. Specifically, inspired by the recent success of Legendre Memory Units (LMU) in sequence learning tasks, we propose LMUFormer, which augments the LMU with convolutional patch embedding and convolutional channel mixer. Moreover, we present a spiking version of this architecture, which introduces the benefit of states within the patch embedding and channel mixer modules while simultaneously reducing the computing complexity. We evaluated our architectures on multiple sequence datasets. In comparison to SOTA transformer-based models within the ANN domain on the SCv2 dataset, our LMUFormer demonstrates comparable performance while necessitating a remarkable 53 times reduction in parameters and a substantial 65 times decrement in FLOPs. Additionally, owing to our model's proficiency in real-time data processing, we can achieve a 32.03% reduction in sequence length, all while incurring an inconsequential decline in performance. Our code is publicly available at https://github.com/zeyuliu1037/LMUFormer.git.

Figures (4)

• Figure 1: The overall network architectures of LMUFormer.
• Figure 2: (a) The structure of the convolutional patch embedding; (b) The delay analysis of the convolutional patch embedding.
• Figure 3: The network architectures of spiking LMU block and the details within the spiking LMU cell.
• Figure 4: Plot of test accuracy as the number of samples in the sequence increases.