Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TsetlinWiSARD: On-Chip Training of Weightless Neural Networks using Tsetlin Automata on FPGAs

Shengyu Duan, Marcos L. L. Sartori, Rishad Shafik, Alex Yakovlev

Abstract

Increasing demands for adaptability, privacy, and security at the edge have persistently pushed the frontiers for a new generation of machine learning (ML) algorithms with training and inference capabilities on-chip. Weightless Neural Network (WNN) is such an algorithm that is principled on lookup table based simple neuron structures. As a result, it offers architectural benefits, such as low-latency, low-complexity inference, compared to deep neural networks that depend heavily on multiply-accumulate operations. However, traditional WNNs rely on memorization-based one-shot training, which either leads to overfitting and reduced accuracy or requires tedious post-training adjustments, limiting their effectiveness for efficient on chip training. In this work, we propose TsetlinWiSARD, a training approach for WNNs that leverages Tsetlin Automata (TAs) to enable probabilistic, feedback-driven learning. It overcomes the overfitting of WiSARD's one-shot training with iterative optimization, while maintaining simple, continuous binary feedback for efficient on-chip training. Central to our approach is a field programmable gate array (FPGA)-based training architecture that delivers state-of-the-art accuracy while significantly improving hardware efficiency. Our approach provides over 1000x faster training when compared with the traditional WiSARD implementation of WNNs. Further, we demonstrate 22% reduced resource usage, 93.3% lower latency, and 64.2% lower power consumption compared to FPGA-based training accelerators implementing other ML algorithms.

TsetlinWiSARD: On-Chip Training of Weightless Neural Networks using Tsetlin Automata on FPGAs

Abstract

Increasing demands for adaptability, privacy, and security at the edge have persistently pushed the frontiers for a new generation of machine learning (ML) algorithms with training and inference capabilities on-chip. Weightless Neural Network (WNN) is such an algorithm that is principled on lookup table based simple neuron structures. As a result, it offers architectural benefits, such as low-latency, low-complexity inference, compared to deep neural networks that depend heavily on multiply-accumulate operations. However, traditional WNNs rely on memorization-based one-shot training, which either leads to overfitting and reduced accuracy or requires tedious post-training adjustments, limiting their effectiveness for efficient on chip training. In this work, we propose TsetlinWiSARD, a training approach for WNNs that leverages Tsetlin Automata (TAs) to enable probabilistic, feedback-driven learning. It overcomes the overfitting of WiSARD's one-shot training with iterative optimization, while maintaining simple, continuous binary feedback for efficient on-chip training. Central to our approach is a field programmable gate array (FPGA)-based training architecture that delivers state-of-the-art accuracy while significantly improving hardware efficiency. Our approach provides over 1000x faster training when compared with the traditional WiSARD implementation of WNNs. Further, we demonstrate 22% reduced resource usage, 93.3% lower latency, and 64.2% lower power consumption compared to FPGA-based training accelerators implementing other ML algorithms.
Paper Structure (12 sections, 11 figures, 1 table)

This paper contains 12 sections, 11 figures, 1 table.

Table of Contents

  1. Introduction
  2. Weightless Neural Networks
  3. TsetlinWiSARD and On-Chip Training
  4. Training Mechanism
  5. Impact of Learning Parameters
  6. Number of TAs
  7. Number of TA states
  8. Feedback probability
  9. On-Chip Training Architecture on FPGA
  10. Off-Chip Training Results
  11. On-Chip Training Results
  12. Conclusion

Figures (11)

  • Figure 1: Inference structures of WiSARD. WiSARD operates on Boolean input features, typically derived by thresholding raw data, and performs classification through a voting mechanism. It needs one discriminator per class. The voting units are LUTs, each receiving a fixed feature subset randomly mapped from the full feature set during training initialization and capable of representing any Boolean function over that subset. All discriminators use the same feature shuffle.
  • Figure 2: Organization of TAs for a LUT.
  • Figure 3: Accuracy vs. LUTs per discriminator (6-input each).
  • Figure 4: Accuracy vs. inputs per LUT.
  • Figure 5: Best accuracy over 50 epochs with varying TA states (300 6-input LUTs per class).
  • ...and 6 more figures