Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Unified Neural Decoding of Perceived, Spoken and Imagined Speech from EEG Signals

Jung-Sun Lee, Ha-Na Jo, Seo-Hyun Lee

TL;DR

This research investigated the effectiveness of deep learning models for non-invasive-based neural signal decoding, with an emphasis on distinguishing between different speech paradigms, including perceived, overt, whispered, and imagined speech, across multiple frequency bands.

Abstract

Brain signals accompany various information relevant to human actions and mental imagery, making them crucial to interpreting and understanding human intentions. Brain-computer interface technology leverages this brain activity to generate external commands for controlling the environment, offering critical advantages to individuals with paralysis or locked-in syndrome. Within the brain-computer interface domain, brain-to-speech research has gained attention, focusing on the direct synthesis of audible speech from brain signals. Most current studies decode speech from brain activity using invasive techniques and emphasize spoken speech data. However, humans express various speech states, and distinguishing these states through non-invasive approaches remains a significant yet challenging task. This research investigated the effectiveness of deep learning models for non-invasive-based neural signal decoding, with an emphasis on distinguishing between different speech paradigms, including perceived, overt, whispered, and imagined speech, across multiple frequency bands. The model utilizing the spatial conventional neural network module demonstrated superior performance compared to other models, especially in the gamma band. Additionally, imagined speech in the theta frequency band, where deep learning also showed strong effects, exhibited statistically significant differences compared to the other speech paradigms.

Towards Unified Neural Decoding of Perceived, Spoken and Imagined Speech from EEG Signals

TL;DR

This research investigated the effectiveness of deep learning models for non-invasive-based neural signal decoding, with an emphasis on distinguishing between different speech paradigms, including perceived, overt, whispered, and imagined speech, across multiple frequency bands.

Abstract

Brain signals accompany various information relevant to human actions and mental imagery, making them crucial to interpreting and understanding human intentions. Brain-computer interface technology leverages this brain activity to generate external commands for controlling the environment, offering critical advantages to individuals with paralysis or locked-in syndrome. Within the brain-computer interface domain, brain-to-speech research has gained attention, focusing on the direct synthesis of audible speech from brain signals. Most current studies decode speech from brain activity using invasive techniques and emphasize spoken speech data. However, humans express various speech states, and distinguishing these states through non-invasive approaches remains a significant yet challenging task. This research investigated the effectiveness of deep learning models for non-invasive-based neural signal decoding, with an emphasis on distinguishing between different speech paradigms, including perceived, overt, whispered, and imagined speech, across multiple frequency bands. The model utilizing the spatial conventional neural network module demonstrated superior performance compared to other models, especially in the gamma band. Additionally, imagined speech in the theta frequency band, where deep learning also showed strong effects, exhibited statistically significant differences compared to the other speech paradigms.

Paper Structure

This paper contains 10 sections, 2 equations, 1 figure, 1 table.

Table of Contents

  1. INTRODUCTION
  2. MATERIALS AND METHODS
  3. Dataset Description
  4. Experimental Setup
  5. Signal Preprocessing
  6. Connectivity Decoding Model
  7. RESULTS AND DISCUSSION
  8. Connectivity Decoding Performance
  9. Statistical Evaluation of Speech Paradigms
  10. CONCLUSION

Figures (1)

  • Figure 1: (a) Placement of 128 electrodes in the 10-5 international standard system. (b) Experimental paradigm for recording EEG signals during four speech states in words. Following the cue, a 1.5-second interval is allocated for perceived speech, during which the participant listens to an auditory stimulus before transitioning to subsequent tasks.