Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dual-TSST: A Dual-Branch Temporal-Spectral-Spatial Transformer Model for EEG Decoding

Hongqi Li, Haodong Zhang, Yitong Chen

TL;DR

This work tackles EEG-based intent decoding by introducing Dual-TSST, a dual-branch CNN-Transformer architecture that fuses temporal-spatial features from raw EEG with temporal-spectral-spatial features from Morlet wavelet time-frequency representations. The dual branches feed into a Transformer-based fusion module, followed by a GAP-MLP classifier, and the model is evaluated on BCI IV 2a, BCI IV 2b, and SEED datasets, achieving average accuracies of 80.67%, 88.64%, and 96.65%, respectively. Ablation studies confirm that both branches and the Transformer contribute substantially to performance, with data augmentation further boosting results. The approach offers a generalizable, high-performance framework for EEG decoding with potential applicability to CNN-Transformer based BCIs and multi-view feature fusion paradigms.

Abstract

The decoding of electroencephalography (EEG) signals allows access to user intentions conveniently, which plays an important role in the fields of human-machine interaction. To effectively extract sufficient characteristics of the multichannel EEG, a novel decoding architecture network with a dual-branch temporal-spectral-spatial transformer (Dual-TSST) is proposed in this study. Specifically, by utilizing convolutional neural networks (CNNs) on different branches, the proposed processing network first extracts the temporal-spatial features of the original EEG and the temporal-spectral-spatial features of time-frequency domain data converted by wavelet transformation, respectively. These perceived features are then integrated by a feature fusion block, serving as the input of the transformer to capture the global long-range dependencies entailed in the non-stationary EEG, and being classified via the global average pooling and multi-layer perceptron blocks. To evaluate the efficacy of the proposed approach, the competitive experiments are conducted on three publicly available datasets of BCI IV 2a, BCI IV 2b, and SEED, with the head-to-head comparison of more than ten other state-of-the-art methods. As a result, our proposed Dual-TSST performs superiorly in various tasks, which achieves the promising EEG classification performance of average accuracy of 80.67% in BCI IV 2a, 88.64% in BCI IV 2b, and 96.65% in SEED, respectively. Extensive ablation experiments conducted between the Dual-TSST and comparative baseline model also reveal the enhanced decoding performance with each module of our proposed method. This study provides a new approach to high-performance EEG decoding, and has great potential for future CNN-Transformer based applications.

Dual-TSST: A Dual-Branch Temporal-Spectral-Spatial Transformer Model for EEG Decoding

TL;DR

This work tackles EEG-based intent decoding by introducing Dual-TSST, a dual-branch CNN-Transformer architecture that fuses temporal-spatial features from raw EEG with temporal-spectral-spatial features from Morlet wavelet time-frequency representations. The dual branches feed into a Transformer-based fusion module, followed by a GAP-MLP classifier, and the model is evaluated on BCI IV 2a, BCI IV 2b, and SEED datasets, achieving average accuracies of 80.67%, 88.64%, and 96.65%, respectively. Ablation studies confirm that both branches and the Transformer contribute substantially to performance, with data augmentation further boosting results. The approach offers a generalizable, high-performance framework for EEG decoding with potential applicability to CNN-Transformer based BCIs and multi-view feature fusion paradigms.

Abstract

The decoding of electroencephalography (EEG) signals allows access to user intentions conveniently, which plays an important role in the fields of human-machine interaction. To effectively extract sufficient characteristics of the multichannel EEG, a novel decoding architecture network with a dual-branch temporal-spectral-spatial transformer (Dual-TSST) is proposed in this study. Specifically, by utilizing convolutional neural networks (CNNs) on different branches, the proposed processing network first extracts the temporal-spatial features of the original EEG and the temporal-spectral-spatial features of time-frequency domain data converted by wavelet transformation, respectively. These perceived features are then integrated by a feature fusion block, serving as the input of the transformer to capture the global long-range dependencies entailed in the non-stationary EEG, and being classified via the global average pooling and multi-layer perceptron blocks. To evaluate the efficacy of the proposed approach, the competitive experiments are conducted on three publicly available datasets of BCI IV 2a, BCI IV 2b, and SEED, with the head-to-head comparison of more than ten other state-of-the-art methods. As a result, our proposed Dual-TSST performs superiorly in various tasks, which achieves the promising EEG classification performance of average accuracy of 80.67% in BCI IV 2a, 88.64% in BCI IV 2b, and 96.65% in SEED, respectively. Extensive ablation experiments conducted between the Dual-TSST and comparative baseline model also reveal the enhanced decoding performance with each module of our proposed method. This study provides a new approach to high-performance EEG decoding, and has great potential for future CNN-Transformer based applications.
Paper Structure (19 sections, 20 equations, 10 figures, 4 tables)

This paper contains 19 sections, 20 equations, 10 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Contribution and Overview
  4. Approach of Dual-TSST Network
  5. Overall Model Architecture
  6. Source of Data
  7. Feature Extraction based on Dual CNN Branches
  8. Feature Fusion based on Transformer
  9. Classification Module
  10. Dataset and Experimental Setup
  11. Datasets
  12. Experiment Setting
  13. Choice of Model Parameters
  14. Results
  15. Head-to-head Comparison Results
  16. ...and 4 more sections

Figures (10)

  • Figure 1: The proposed Dual-TSST framework, including feature extraction of two CNN branches, feature fusion with Transformer, and classification modules.
  • Figure 2: Structure and data flow of Branch I for feature extraction module.
  • Figure 3: Structure and data flow of Branch II for feature extraction module.
  • Figure 4: The influence of pointwise dimension on model performance.
  • Figure 5: The influence of the Transformer layer number on the average accuracy
  • ...and 5 more figures