CognitionCapturer: Decoding Visual Stimuli From Human EEG Signal With Multimodal Information

TL;DR

CognitionCapturer addresses the challenge of decoding visual stimuli from EEG by exploiting multimodal information beyond the image modality. It introduces Modality Expert Encoders to align EEG with image, text, and depth modalities via contrastive learning, and employs a diffusion prior to map EEG embeddings into CLIP space for reconstruction with pretrained SDXL-IP-Adapters, without fine-tuning generators. The approach yields state-of-the-art classification and improved visual reconstructions, with ablations and Grad-CAM analyses demonstrating complementary information across modalities and interpretable brain-region engagement. The work advances practical brain decoding by enabling scalable, multimodal EEG representations that capture both semantic and structural aspects of visual stimuli, with potential impact on real-time neural decoding, neurometric interfaces, and cognitive neuroscience research.

Abstract

Electroencephalogram (EEG) signals have attracted significant attention from researchers due to their non-invasive nature and high temporal sensitivity in decoding visual stimuli. However, most recent studies have focused solely on the relationship between EEG and image data pairs, neglecting the valuable ``beyond-image-modality" information embedded in EEG signals. This results in the loss of critical multimodal information in EEG. To address this limitation, we propose CognitionCapturer, a unified framework that fully leverages multimodal data to represent EEG signals. Specifically, CognitionCapturer trains Modality Expert Encoders for each modality to extract cross-modal information from the EEG modality. Then, it introduces a diffusion prior to map the EEG embedding space to the CLIP embedding space, followed by using a pretrained generative model, the proposed framework can reconstruct visual stimuli with high semantic and structural fidelity. Notably, the framework does not require any fine-tuning of the generative models and can be extended to incorporate more modalities. Through extensive experiments, we demonstrate that CognitionCapturer outperforms state-of-the-art methods both qualitatively and quantitatively. Code: https://github.com/XiaoZhangYES/CognitionCapturer.

Figures (11)

• Figure 1: We believe that for image-EEG pairs, relying solely on the mutual information between images and EEG signals can lead to underutilization of EEG information. To address this issue, we utilize multimodal information to capture meaningful information in the EEG signals. The dashed lines in the figure below illustrate some of our successful reconstruction results.
• Figure 2: Overall framework of CognitionCapturer. 1: In the contrastive learning stage, different EEG-Modality data pairs are fed into different Modality Expert Encoders for processing. The embeddings obtained from the contrastive learning stage can be used for various downstream tasks. 2: To use pre-trained image generation models, we apply a Diffusion Prior model to map the EEG embeddings into CLIP space while retaining their original information. 3: Using pre-trained SDXL and IP-Adapters with different structures, we integrate the EEG embeddings from different modalities to reconstruct visual stimuli.
• Figure 3: Visual Comparison. Selected reconstruction results from subject-08 show that our reconstructed visual stimuli exhibit finer-grained features.
• Figure 4: Reconstruction results of CognitionCapturer on different modality and comparison with prior work.
• Figure 5: (A) The Grad-CAM results from different Modality Expert Encoders show the activation in the occipital and temporal lobes related to the input EEG signals. (B) The Grad-CAM results from different modality Expert Encoders on the brain signals corresponding to the example image, visualizing the regions of attention in the images and comparing them with the original CLIP embeddings.