SI-SD: Sleep Interpreter through awake-guided cross-subject Semantic Decoding

TL;DR

The paper tackles decoding semantic content from human sleep EEG by proposing SI-SD, a Transformer-based framework that aligns awake neural latent sequences with sleep representations. It introduces an awake-guided latent alignment with a three-pathway architecture and a contrastive objective to produce subject-agnostic sleep decoders, validated on a large 134-subject dataset with image-, audio-, and TMR-related sleep data. The approach achieves state-of-the-art unseen-subject accuracy (up to around 40% in certain SO-related groups and around 30% after fine-tuning) for 15-way sleep semantics in NREM 2/3 and REM sleep, and reveals that Slow Oscillation events and spindle coupling modulate decoding performance. By releasing the dataset and providing reproducible code, the work establishes a neuro-inspired AI framework for sleep decoding with meaningful implications for memory research and brain-computer interface applications.

Abstract

Understanding semantic content from brain activity during sleep represents a major goal in neuroscience. While studies in rodents have shown spontaneous neural reactivation of memories during sleep, capturing the semantic content of human sleep poses a significant challenge due to the absence of well-annotated sleep datasets and the substantial differences in neural patterns between wakefulness and sleep. To address these challenges, we designed a novel cognitive neuroscience experiment and collected a comprehensive, well-annotated electroencephalography (EEG) dataset from 134 subjects during both wakefulness and sleep. Leveraging this benchmark dataset, we developed SI-SD that enhances sleep semantic decoding through the position-wise alignment of neural latent sequence between wakefulness and sleep. In the 15-way classification task, our model achieves 24.12% and 21.39% top-1 accuracy on unseen subjects for NREM 2/3 and REM sleep, respectively, surpassing all other baselines. With additional fine-tuning, decoding performance improves to 30.32% and 31.65%, respectively. Besides, inspired by previous neuroscientific findings, we systematically analyze how the "Slow Oscillation" event impacts decoding performance in NREM 2/3 sleep -- decoding performance on unseen subjects further improves to 40.02%. Together, our findings and methodologies contribute to a promising neuro-AI framework for decoding brain activity during sleep.

Figures (21)

• Figure 1: The experiment design for sleep semantic decoding. During the experiment, each subject must memorize the shared set of semantic concepts (including 15 pre-determined semantic concepts, each paired with both an image stimulus and an audio stimulus); see Appendix \ref{['sec:supp-expr-design']} for more details.
• Figure 2: The overall architecture of Neural Transformer.
• Figure 3: The overall architecture of SI-SD.
• Figure 4: The analysis related to the "Slow Oscillation" event.(a). The stages of the "Slow Oscillation" event. We divide NREM 2/3 sleep samples into 8 groups according to the onset of their audio stimuli. (b). The decoding performance of different SO-groups during NREM 2/3 sleep.
• Figure 5: The experiment pipeline of each trial during the function localizer stage. Here, we only illustrate the pipeline for the image-audio task. For the audio-image task, we simply reverse the presentation order of the image and audio stimuli.