Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Transfer Learning for Covert Speech Classification Using EEG Hilbert Envelope and Temporal Fine Structure

Saravanakumar Duraisamy, Mateusz Dubiel, Maurice Rekrut, Luis A. Leiva

TL;DR

This work tackles the challenge of decoding covert speech from noninvasive EEG by reducing the training burden through transfer learning. It uses Hilbert Envelope (ENV) and Temporal Fine Structure (TFS) features extracted from EEG and trains a BiLSTM on overt speech data, then transfers the model to covert (imagined) speech. The approach achieves an average overt accuracy of 86.44% and a covert accuracy of 79.82% when using the overt-trained classifier, with performance improving when modest amounts of covert data are included for fine-tuning. By demonstrating effective cross-modal transfer and reporting high accuracies, the study advances practical imagined-speech BCIs for assistive communication, addressing data collection and onset-uncertainty challenges.

Abstract

Brain-Computer Interfaces (BCIs) can decode imagined speech from neural activity. However, these systems typically require extensive training sessions where participants imaginedly repeat words, leading to mental fatigue and difficulties identifying the onset of words, especially when imagining sequences of words. This paper addresses these challenges by transferring a classifier trained in overt speech data to covert speech classification. We used electroencephalogram (EEG) features derived from the Hilbert envelope and temporal fine structure, and used them to train a bidirectional long-short-term memory (BiLSTM) model for classification. Our method reduces the burden of extensive training and achieves state-of-the-art classification accuracy: 86.44% for overt speech and 79.82% for covert speech using the overt speech classifier.

Transfer Learning for Covert Speech Classification Using EEG Hilbert Envelope and Temporal Fine Structure

TL;DR

This work tackles the challenge of decoding covert speech from noninvasive EEG by reducing the training burden through transfer learning. It uses Hilbert Envelope (ENV) and Temporal Fine Structure (TFS) features extracted from EEG and trains a BiLSTM on overt speech data, then transfers the model to covert (imagined) speech. The approach achieves an average overt accuracy of 86.44% and a covert accuracy of 79.82% when using the overt-trained classifier, with performance improving when modest amounts of covert data are included for fine-tuning. By demonstrating effective cross-modal transfer and reporting high accuracies, the study advances practical imagined-speech BCIs for assistive communication, addressing data collection and onset-uncertainty challenges.

Abstract

Brain-Computer Interfaces (BCIs) can decode imagined speech from neural activity. However, these systems typically require extensive training sessions where participants imaginedly repeat words, leading to mental fatigue and difficulties identifying the onset of words, especially when imagining sequences of words. This paper addresses these challenges by transferring a classifier trained in overt speech data to covert speech classification. We used electroencephalogram (EEG) features derived from the Hilbert envelope and temporal fine structure, and used them to train a bidirectional long-short-term memory (BiLSTM) model for classification. Our method reduces the burden of extensive training and achieves state-of-the-art classification accuracy: 86.44% for overt speech and 79.82% for covert speech using the overt speech classifier.

Paper Structure

This paper contains 15 sections, 3 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Experimental design and procedure
  5. Participants
  6. Design
  7. Procedure
  8. Data collection
  9. Data preprocessing and feature extraction
  10. Dataset splits and model description
  11. Results and discussion
  12. Covert-Overt EEG decoding
  13. Classification results
  14. Limitations and Future Work
  15. Conclusion

Figures (3)

  • Figure 1: Protocol for collecting imagined speech data mohamed2024speech. A spacebar press triggers the interaction during overt speech, while imagined speech interactions occur in separate sessions.
  • Figure 2: ENV and TFS of the speech command Right.
  • Figure 3: BiLSTM model performance with varying data splits of covert EEG data. Error bars denote SDs.