Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MindCine: Multimodal EEG-to-Video Reconstruction with Large-Scale Pretrained Models

Tian-Yi Zhou, Xuan-Hao Liu, Bao-Liang Lu, Wei-Long Zheng

TL;DR

This paper tackles EEG-to-video reconstruction, addressing the limitations of single-modality text alignment and scarce training data. It introduces MindCine, a two-branch framework comprising a Semantic Decoding Module with multimodal joint learning and a Large-scale Pretrained EEG encoder, and a Perceptual Decoding Module featuring EmbedNet and a CausalSeq Transformer, all guided by a diffusion-based inference backbone. The approach achieves state-of-the-art results on the SEED-DV benchmark across semantic and pixel-level metrics, and ablation experiments show that both decoding modules and multimodal conditioning are crucial. By leveraging large-scale EEG models and multimodal priors, MindCine demonstrates robust EEG-to-video reconstruction under limited data, highlighting the value of rich EEG representations and cross-modal alignment for understanding dynamic visual perception.

Abstract

Reconstructing human dynamic visual perception from electroencephalography (EEG) signals is of great research significance since EEG's non-invasiveness and high temporal resolution. However, EEG-to-video reconstruction remains challenging due to: 1) Single Modality: existing studies solely align EEG signals with the text modality, which ignores other modalities and are prone to suffer from overfitting problems; 2) Data Scarcity: current methods often have difficulty training to converge with limited EEG-video data. To solve the above problems, we propose a novel framework MindCine to achieve high-fidelity video reconstructions on limited data. We employ a multimodal joint learning strategy to incorporate beyond-text modalities in the training stage and leverage a pre-trained large EEG model to relieve the data scarcity issue for decoding semantic information, while a Seq2Seq model with causal attention is specifically designed for decoding perceptual information. Extensive experiments demonstrate that our model outperforms state-of-the-art methods both qualitatively and quantitatively. Additionally, the results underscore the effectiveness of the complementary strengths of different modalities and demonstrate that leveraging a large-scale EEG model can further enhance reconstruction performance by alleviating the challenges associated with limited data.

MindCine: Multimodal EEG-to-Video Reconstruction with Large-Scale Pretrained Models

TL;DR

This paper tackles EEG-to-video reconstruction, addressing the limitations of single-modality text alignment and scarce training data. It introduces MindCine, a two-branch framework comprising a Semantic Decoding Module with multimodal joint learning and a Large-scale Pretrained EEG encoder, and a Perceptual Decoding Module featuring EmbedNet and a CausalSeq Transformer, all guided by a diffusion-based inference backbone. The approach achieves state-of-the-art results on the SEED-DV benchmark across semantic and pixel-level metrics, and ablation experiments show that both decoding modules and multimodal conditioning are crucial. By leveraging large-scale EEG models and multimodal priors, MindCine demonstrates robust EEG-to-video reconstruction under limited data, highlighting the value of rich EEG representations and cross-modal alignment for understanding dynamic visual perception.

Abstract

Reconstructing human dynamic visual perception from electroencephalography (EEG) signals is of great research significance since EEG's non-invasiveness and high temporal resolution. However, EEG-to-video reconstruction remains challenging due to: 1) Single Modality: existing studies solely align EEG signals with the text modality, which ignores other modalities and are prone to suffer from overfitting problems; 2) Data Scarcity: current methods often have difficulty training to converge with limited EEG-video data. To solve the above problems, we propose a novel framework MindCine to achieve high-fidelity video reconstructions on limited data. We employ a multimodal joint learning strategy to incorporate beyond-text modalities in the training stage and leverage a pre-trained large EEG model to relieve the data scarcity issue for decoding semantic information, while a Seq2Seq model with causal attention is specifically designed for decoding perceptual information. Extensive experiments demonstrate that our model outperforms state-of-the-art methods both qualitatively and quantitatively. Additionally, the results underscore the effectiveness of the complementary strengths of different modalities and demonstrate that leveraging a large-scale EEG model can further enhance reconstruction performance by alleviating the challenges associated with limited data.
Paper Structure (22 sections, 7 equations, 5 figures, 2 tables)

This paper contains 22 sections, 7 equations, 5 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. METHOD
  3. Semantic Decoding Module
  4. Large-scale Pretrained Encoder
  5. Multimodal Joint Learning
  6. Semantic Predictor
  7. Loss Function Co-optimization
  8. Perceptual Decoding Module
  9. EmbedNet
  10. CausalSeq Model
  11. Inference Module
  12. EXPERIMENT
  13. Datasets
  14. Evaluation Metrics
  15. RESULTS
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Brain Decoding Paradigms: Previous vs. Ours.
  • Figure 2: The overall framework of MindCine.
  • Figure 3: Successful reconstruction results on SEED-DV dataset. Our MindCine reconstructs videos with higher quality and more precise semantics.
  • Figure 4: Successful reconstruction results with different large EEG models
  • Figure 5: Successful video reconstruction results using different combinations of modalities.