Threshold Neuron: A Brain-inspired Artificial Neuron for Efficient On-device Inference
Zihao Zheng, Yuanchun Li, Jiayu Chen, Peng Zhou, Xiang Chen, Yunxin Liu
TL;DR
This paper introduces Threshold Neurons, a multiplication-free, brain-inspired artificial neuron that uses a learnable threshold and subtraction-based interactions to replace traditional weighted sums. By unifying neuron types and removing normalization, rectification, and pooling, the authors build Threshold-Net, which remains compatible with existing architectures and supports Multiplication Injection (MI) to blend with conventional layers. They provide GPU-based training and FPGA-based inference demonstrations, reporting substantial kernel-level area and power savings (3.89x–4.33x area; 7.51x–8.19x power) and system-level FPGA gains (2.52x power, 1.75x latency). Across image classification, diffusion-based generation, and sensing tasks, Threshold-Net achieves strong accuracy with significantly reduced multiplications and hardware complexity, illustrating a practical path toward efficient on-device AI. The work also outlines future directions toward NAS-driven optimization and ASIC development to further harness the simplicity and efficiency of Threshold Neurons.
Abstract
Enhancing the computational efficiency of on-device Deep Neural Networks (DNNs) remains a significant challengein mobile and edge computing. As we aim to execute increasingly complex tasks with constrained computational resources, much of the research has focused on compressing neural network structures and optimizing systems. Although many studies have focused on compressing neural network structures and parameters or optimizing underlying systems, there has been limited attention on optimizing the fundamental building blocks of neural networks: the neurons. In this study, we deliberate on a simple but important research question: Can we design artificial neurons that offer greater efficiency than the traditional neuron paradigm? Inspired by the threshold mechanisms and the excitation-inhibition balance observed in biological neurons, we propose a novel artificial neuron model, Threshold Neurons. Using Threshold Neurons, we can construct neural networks similar to those with traditional artificial neurons, while significantly reducing hardware implementation complexity. Our extensive experiments validate the effectiveness of neural networks utilizing Threshold Neurons, achieving substantial power savings of 7.51x to 8.19x and area savings of 3.89x to 4.33x at the kernel level, with minimal loss in precision. Furthermore, FPGA-based implementations of these networks demonstrate 2.52x power savings and 1.75x speed enhancements at the system level. The source code will be made available upon publication.