Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scheduled Knowledge Acquisition on Lightweight Vector Symbolic Architectures for Brain-Computer Interfaces

Yejia Liu, Shijin Duan, Xiaolin Xu, Shaolei Ren

TL;DR

This work proposes a simple scheduled knowledge distillation method based on curriculum data order to enable the student to gradually build knowledge from the teacher model, controlled by an $\alpha$ scheduler and employs the LDC/VSA as the student model to enhance the on-device inference efficiency for tiny BCI devices that demand low latency.

Abstract

Brain-Computer interfaces (BCIs) are typically designed to be lightweight and responsive in real-time to provide users timely feedback. Classical feature engineering is computationally efficient but has low accuracy, whereas the recent neural networks (DNNs) improve accuracy but are computationally expensive and incur high latency. As a promising alternative, the low-dimensional computing (LDC) classifier based on vector symbolic architecture (VSA), achieves small model size yet higher accuracy than classical feature engineering methods. However, its accuracy still lags behind that of modern DNNs, making it challenging to process complex brain signals. To improve the accuracy of a small model, knowledge distillation is a popular method. However, maintaining a constant level of distillation between the teacher and student models may not be the best way for a growing student during its progressive learning stages. In this work, we propose a simple scheduled knowledge distillation method based on curriculum data order to enable the student to gradually build knowledge from the teacher model, controlled by an $α$ scheduler. Meanwhile, we employ the LDC/VSA as the student model to enhance the on-device inference efficiency for tiny BCI devices that demand low latency. The empirical results have demonstrated that our approach achieves better tradeoff between accuracy and hardware efficiency compared to other methods.

Scheduled Knowledge Acquisition on Lightweight Vector Symbolic Architectures for Brain-Computer Interfaces

TL;DR

This work proposes a simple scheduled knowledge distillation method based on curriculum data order to enable the student to gradually build knowledge from the teacher model, controlled by an scheduler and employs the LDC/VSA as the student model to enhance the on-device inference efficiency for tiny BCI devices that demand low latency.

Abstract

Brain-Computer interfaces (BCIs) are typically designed to be lightweight and responsive in real-time to provide users timely feedback. Classical feature engineering is computationally efficient but has low accuracy, whereas the recent neural networks (DNNs) improve accuracy but are computationally expensive and incur high latency. As a promising alternative, the low-dimensional computing (LDC) classifier based on vector symbolic architecture (VSA), achieves small model size yet higher accuracy than classical feature engineering methods. However, its accuracy still lags behind that of modern DNNs, making it challenging to process complex brain signals. To improve the accuracy of a small model, knowledge distillation is a popular method. However, maintaining a constant level of distillation between the teacher and student models may not be the best way for a growing student during its progressive learning stages. In this work, we propose a simple scheduled knowledge distillation method based on curriculum data order to enable the student to gradually build knowledge from the teacher model, controlled by an scheduler. Meanwhile, we employ the LDC/VSA as the student model to enhance the on-device inference efficiency for tiny BCI devices that demand low latency. The empirical results have demonstrated that our approach achieves better tradeoff between accuracy and hardware efficiency compared to other methods.
Paper Structure (14 sections, 1 equation, 3 figures, 3 tables, 1 algorithm)

This paper contains 14 sections, 1 equation, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Student: Low-dimensional Vector Symbolic Architecture
  4. Hyperdimensional Computing/Vector Symbolic Architecture (HDC/VSA)
  5. Low-dimensional Computing (LDC) Classifier
  6. Scheduling the Knowledge Distillation: ScheduledKD-LDC
  7. Experiments
  8. Experimental Setup
  9. Main Results
  10. Analysis of the Scheduler
  11. Efficacy of Curriculum Data Order
  12. Related Works
  13. Discussion and Future Works
  14. Acknowledgment

Figures (3)

  • Figure 1: Comparison of accuracy and inference efficiency of various methods on the Motor Imagery dataset. The ScheduledKD-LDC has achieved a good tradeoff between accuracy and efficiency, considering the model size and the number of FPMAC operations.
  • Figure 2: Overview of the ScheduledKD-LDC. We use an $\alpha$ scheduler and curriculum data order to enable the student model to gradually distill knowledge from the complex teacher model. The student model, low-dimensional computing classifier (LDC), is based on the vector symbolic architecture for efficient on-device inference.
  • Figure 3: Comparison of different $\alpha$ setups, without employing data curriculum in the KD setting on the Motor Imagery, X11 and S4b datasets. (a), (b): With different temperatures $\tau$. (c), (d): With different starting values of $\alpha$.