Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neural-MCRL: Neural Multimodal Contrastive Representation Learning for EEG-based Visual Decoding

Yueyang Li, Zijian Kang, Shengyu Gong, Wenhao Dong, Weiming Zeng, Hongjie Yan, Wai Ting Siok, Nizhuan Wang

TL;DR

Neural-MCRL tackles the challenge of decoding visual representations from EEG by addressing semantic gaps in existing multimodal contrastive learning. It introduces NESTA, a subject-specific EEG encoder with adaptive spectral processing, and a semantic enhancement pipeline plus EITRA that anchors EEG, image, and text into a shared semantic space through semantic-guided attention and cross-modal alignment. The approach achieves state-of-the-art zero-shot performance and cross-subject generalization on THINGS-EEG, with ablations confirming the importance of subject-specific transformations, intra-/inter-modal semantic completion, and semantic-guided alignment. Together, these components advance EEG-based neural visual decoding for BMI by producing more consistent, interpretable, and generalizable neural-semantic representations.

Abstract

Decoding neural visual representations from electroencephalogram (EEG)-based brain activity is crucial for advancing brain-machine interfaces (BMI) and has transformative potential for neural sensory rehabilitation. While multimodal contrastive representation learning (MCRL) has shown promise in neural decoding, existing methods often overlook semantic consistency and completeness within modalities and lack effective semantic alignment across modalities. This limits their ability to capture the complex representations of visual neural responses. We propose Neural-MCRL, a novel framework that achieves multimodal alignment through semantic bridging and cross-attention mechanisms, while ensuring completeness within modalities and consistency across modalities. Our framework also features the Neural Encoder with Spectral-Temporal Adaptation (NESTA), a EEG encoder that adaptively captures spectral patterns and learns subject-specific transformations. Experimental results demonstrate significant improvements in visual decoding accuracy and model generalization compared to state-of-the-art methods, advancing the field of EEG-based neural visual representation decoding in BMI. Codes will be available at: https://github.com/NZWANG/Neural-MCRL.

Neural-MCRL: Neural Multimodal Contrastive Representation Learning for EEG-based Visual Decoding

TL;DR

Neural-MCRL tackles the challenge of decoding visual representations from EEG by addressing semantic gaps in existing multimodal contrastive learning. It introduces NESTA, a subject-specific EEG encoder with adaptive spectral processing, and a semantic enhancement pipeline plus EITRA that anchors EEG, image, and text into a shared semantic space through semantic-guided attention and cross-modal alignment. The approach achieves state-of-the-art zero-shot performance and cross-subject generalization on THINGS-EEG, with ablations confirming the importance of subject-specific transformations, intra-/inter-modal semantic completion, and semantic-guided alignment. Together, these components advance EEG-based neural visual decoding for BMI by producing more consistent, interpretable, and generalizable neural-semantic representations.

Abstract

Decoding neural visual representations from electroencephalogram (EEG)-based brain activity is crucial for advancing brain-machine interfaces (BMI) and has transformative potential for neural sensory rehabilitation. While multimodal contrastive representation learning (MCRL) has shown promise in neural decoding, existing methods often overlook semantic consistency and completeness within modalities and lack effective semantic alignment across modalities. This limits their ability to capture the complex representations of visual neural responses. We propose Neural-MCRL, a novel framework that achieves multimodal alignment through semantic bridging and cross-attention mechanisms, while ensuring completeness within modalities and consistency across modalities. Our framework also features the Neural Encoder with Spectral-Temporal Adaptation (NESTA), a EEG encoder that adaptively captures spectral patterns and learns subject-specific transformations. Experimental results demonstrate significant improvements in visual decoding accuracy and model generalization compared to state-of-the-art methods, advancing the field of EEG-based neural visual representation decoding in BMI. Codes will be available at: https://github.com/NZWANG/Neural-MCRL.

Paper Structure

This paper contains 16 sections, 15 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. EEG-based Visual Decoding
  4. Multimodal Contrastive Representation Learning
  5. Method
  6. Problem Formulation
  7. NESTA for EEG Embedding
  8. Image Encoder
  9. Semantic Enhancement
  10. EEG-Image-Text Representation Alignment (EITRA)
  11. Experiments and Results
  12. Experimental Details
  13. Overall Performance and Ablation Studies
  14. Semantic Similarity and Visualization Analysis
  15. Performance Comparison across EEG Encoders
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Overall framework of the Neural-MCRL.
  • Figure 2: Architecture of the NESTA for EEG embedding.
  • Figure 3: Semantic similarity analysis and visualization. (A) Cosine similarity of 200 concepts in the test set. The results calculated by the trained models of 10 subjects were averaged, and all the concepts were rearranged into six categories. (B) Comparison between ground truth (first column) and the top-5 predicted. (C) Visualization in the latent space of EEG/image by t-SNE. (D) Average in-subject accuracy across different EEG Encoders.