EIT-1M: One Million EEG-Image-Text Pairs for Human Visual-textual Recognition and More

TL;DR

EIT-1M introduces a large-scale, multi-modal EEG dataset consisting of over 1 million EEG-image-text epochs collected from five participants while they processed visual and textual stimuli derived from CIFAR-10. The dataset employs RSVP with precise stimulus timing, 64-channel EEG, and block-based repetition to capture robust brain responses to simultaneous modalities. Analyses include EEG topography and ERP assessments, demonstrating modality-specific and shared neural dynamics, and experiments show improved recognition when using combined visual and textual EEG signals, along with EEG-to-visual generation capabilities. This resource aims to advance multi-modal AI and cognitive neuroscience by enabling robust brain decoding and cross-modal generation research, while acknowledging limitations and outlining future expansion opportunities and ethical considerations.

Abstract

Recently, electroencephalography (EEG) signals have been actively incorporated to decode brain activity to visual or textual stimuli and achieve object recognition in multi-modal AI. Accordingly, endeavors have been focused on building EEG-based datasets from visual or textual single-modal stimuli. However, these datasets offer limited EEG epochs per category, and the complex semantics of stimuli presented to participants compromise their quality and fidelity in capturing precise brain activity. The study in neuroscience unveils that the relationship between visual and textual stimulus in EEG recordings provides valuable insights into the brain's ability to process and integrate multi-modal information simultaneously. Inspired by this, we propose a novel large-scale multi-modal dataset, named EIT-1M, with over 1 million EEG-image-text pairs. Our dataset is superior in its capacity of reflecting brain activities in simultaneously processing multi-modal information. To achieve this, we collected data pairs while participants viewed alternating sequences of visual-textual stimuli from 60K natural images and category-specific texts. Common semantic categories are also included to elicit better reactions from participants' brains. Meanwhile, response-based stimulus timing and repetition across blocks and sessions are included to ensure data diversity. To verify the effectiveness of EIT-1M, we provide an in-depth analysis of EEG data captured from multi-modal stimuli across different categories and participants, along with data quality scores for transparency. We demonstrate its validity on two tasks: 1) EEG recognition from visual or textual stimuli or both and 2) EEG-to-visual generation.

Figures (6)

• Figure 1: (a) actiCHamp Plus device. (b) Experimental setup with monitor 80 cm from participant. (c) Injecting conductive gel. (d) Visual stimuli. (e) Textual stimuli. (f) Speech stimuli.
• Figure 2: (a) EEG signals from high-resolution visual stimuli. (b) EEG signals from our visual stimuli.
• Figure 3: Schematic overview of the structure of trials, blocks, categories and sessions with RSVP paradigm. (a) Training set of CIFAR-10 dataset; (b) Testing set of CIFAR-10 dataset.
• Figure 4: EEG topographic maps and corresponding signals averaged over events for the participant viewing visual stimuli (left column) viewing the airplane (1st row) and frog (2nd row) images from the CIFAR-10 dataset, and events for the participant viewing textual stimuli (right column) viewing the airplane (1st row) and frog (2nd row) text.
• Figure 5: ERPs averaged over occipital and parietal electrodes for the participant viewing stimuli from (a) visual images and (b) the category text. Shaded areas around the grand average ERP represent standard deviations at each time point.