Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Scalable Handwriting Communication via EEG Decoding and Latent Embedding Integration

Jun-Young Kim, Deok-Seon Kim, Seo-Hyun Lee

TL;DR

This work incorporates hand kinematics to guide the extraction of the consistent embeddings from high-dimensional neural recordings using auxiliary variables (CEBRA) and demonstrates the feasibility of fine-grained handwriting decoding from EEG signals.

Abstract

In recent years, brain-computer interfaces have made advances in decoding various motor-related tasks, including gesture recognition and movement classification, utilizing electroencephalogram (EEG) data. These developments are fundamental in exploring how neural signals can be interpreted to recognize specific physical actions. This study centers on a written alphabet classification task, where we aim to decode EEG signals associated with handwriting. To achieve this, we incorporate hand kinematics to guide the extraction of the consistent embeddings from high-dimensional neural recordings using auxiliary variables (CEBRA). These CEBRA embeddings, along with the EEG, are processed by a parallel convolutional neural network model that extracts features from both data sources simultaneously. The model classifies nine different handwritten characters, including symbols such as exclamation marks and commas, within the alphabet. We evaluate the model using a quantitative five-fold cross-validation approach and explore the structure of the embedding space through visualizations. Our approach achieves a classification accuracy of 91 % for the nine-class task, demonstrating the feasibility of fine-grained handwriting decoding from EEG.

Towards Scalable Handwriting Communication via EEG Decoding and Latent Embedding Integration

TL;DR

This work incorporates hand kinematics to guide the extraction of the consistent embeddings from high-dimensional neural recordings using auxiliary variables (CEBRA) and demonstrates the feasibility of fine-grained handwriting decoding from EEG signals.

Abstract

In recent years, brain-computer interfaces have made advances in decoding various motor-related tasks, including gesture recognition and movement classification, utilizing electroencephalogram (EEG) data. These developments are fundamental in exploring how neural signals can be interpreted to recognize specific physical actions. This study centers on a written alphabet classification task, where we aim to decode EEG signals associated with handwriting. To achieve this, we incorporate hand kinematics to guide the extraction of the consistent embeddings from high-dimensional neural recordings using auxiliary variables (CEBRA). These CEBRA embeddings, along with the EEG, are processed by a parallel convolutional neural network model that extracts features from both data sources simultaneously. The model classifies nine different handwritten characters, including symbols such as exclamation marks and commas, within the alphabet. We evaluate the model using a quantitative five-fold cross-validation approach and explore the structure of the embedding space through visualizations. Our approach achieves a classification accuracy of 91 % for the nine-class task, demonstrating the feasibility of fine-grained handwriting decoding from EEG.

Paper Structure

This paper contains 11 sections, 2 figures, 1 table.

Table of Contents

  1. INTRODUCTION
  2. MATERIALS AND METHODS
  3. Preparation of EEG and Kinematic Data
  4. Joint Learning: CEBRA Embeddings and EEG Data
  5. Learning CEBRA embeddings
  6. CNN--based feature fusion
  7. Evaluation and visualization
  8. RESULTS AND DISCUSSION
  9. EEG Decoding Performance
  10. Data Embeddings
  11. CONCLUSION

Figures (2)

  • Figure 1: Framework for hand kinematics--aware consistent embedding generation and feature fusion using CEBRA--CNN.
  • Figure 2: (a) PCA projection of the CEBRA embeddings, showing the structure of both continuous and discrete labels. (b) t--SNE visualization comparing the clustering of the preprocessed EEG against the 2--D and 16--D CEBRA embeddings.