Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SEE: Semantically Aligned EEG-to-Text Translation

Yitian Tao, Yan Liang, Luoyu Wang, Yongqing Li, Qing Yang, Han Zhang

TL;DR

SEMantically Aligned EEG-to-Text Translation is proposed, a novel method aimed at improving EEG-to-Text decoding by seamlessly integrating two modules into a pre-trained BART language model, which enhances the feasibility of accurate EEG-to-Text decoding.

Abstract

Decoding neurophysiological signals into language is of great research interest within brain-computer interface (BCI) applications. Electroencephalography (EEG), known for its non-invasiveness, ease of use, and cost-effectiveness, has been a popular method in this field. However, current EEG-to-Text decoding approaches face challenges due to the huge domain gap between EEG recordings and raw texts, inherent data bias, and small closed vocabularies. In this paper, we propose SEE: Semantically Aligned EEG-to-Text Translation, a novel method aimed at improving EEG-to-Text decoding by seamlessly integrating two modules into a pre-trained BART language model. These two modules include (1) a Cross-Modal Codebook that learns cross-modal representations to enhance feature consolidation and mitigate domain gap, and (2) a Semantic Matching Module that fully utilizes pre-trained text representations to align multi-modal features extracted from EEG-Text pairs while considering noise caused by false negatives, i.e., data from different EEG-Text pairs that have similar semantic meanings. Experimental results on the Zurich Cognitive Language Processing Corpus (ZuCo) demonstrate the effectiveness of SEE, which enhances the feasibility of accurate EEG-to-Text decoding.

SEE: Semantically Aligned EEG-to-Text Translation

TL;DR

SEMantically Aligned EEG-to-Text Translation is proposed, a novel method aimed at improving EEG-to-Text decoding by seamlessly integrating two modules into a pre-trained BART language model, which enhances the feasibility of accurate EEG-to-Text decoding.

Abstract

Decoding neurophysiological signals into language is of great research interest within brain-computer interface (BCI) applications. Electroencephalography (EEG), known for its non-invasiveness, ease of use, and cost-effectiveness, has been a popular method in this field. However, current EEG-to-Text decoding approaches face challenges due to the huge domain gap between EEG recordings and raw texts, inherent data bias, and small closed vocabularies. In this paper, we propose SEE: Semantically Aligned EEG-to-Text Translation, a novel method aimed at improving EEG-to-Text decoding by seamlessly integrating two modules into a pre-trained BART language model. These two modules include (1) a Cross-Modal Codebook that learns cross-modal representations to enhance feature consolidation and mitigate domain gap, and (2) a Semantic Matching Module that fully utilizes pre-trained text representations to align multi-modal features extracted from EEG-Text pairs while considering noise caused by false negatives, i.e., data from different EEG-Text pairs that have similar semantic meanings. Experimental results on the Zurich Cognitive Language Processing Corpus (ZuCo) demonstrate the effectiveness of SEE, which enhances the feasibility of accurate EEG-to-Text decoding.
Paper Structure (9 sections, 14 equations, 1 figure, 3 tables)

This paper contains 9 sections, 14 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Cross-Modal Codebook Retrieval
  4. Semantic Matching
  5. Text Decoding
  6. Experiments
  7. Dataset and Metrices
  8. Experimental Analysis
  9. Conclusion

Figures (1)

  • Figure 1: Illustration of SEE, where two modules are seamlessly embedded into a pre-trained transformer-based language model BART as a whole: 1) A Cross-Modal Codebook M which stores cross-modal representations for multi-modal retrieval, thus suggesting feature enhancement and modality bias mitigation; 2) A Semantic Matching module which is capable of aligning multi-modal features while considering the semantic consistency.