Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

sDREAMER: Self-distilled Mixture-of-Modality-Experts Transformer for Automatic Sleep Staging

Jingyuan Chen, Yuan Yao, Mie Anderson, Natalie Hauglund, Celia Kjaerby, Verena Untiet, Maiken Nedergaard, Jiebo Luo

TL;DR

sDREAMER introduces a self-distilled Mixture-of-Modality-Experts transformer for sleep staging that operates on EEG, EMG, or their combination. By employing three parallel modality pathways with partially shared weights and a cross-modal mix pathway, coupled with a self-distillation scheme, it achieves strong performance in both single-channel and multi-channel inference on a mouse dataset. The architecture integrates patch-based input, epoch- and sequence-level MoME transformers, and a KL-divergence based distillation objective to improve cross-modal interactions. Empirical results show clear advantages over baseline transformer and traditional methods, with ablations confirming the value of joint encoding and self-distillation, and visualizations supporting meaningful, aligned latent representations across modalities.

Abstract

Automatic sleep staging based on electroencephalography (EEG) and electromyography (EMG) signals is an important aspect of sleep-related research. Current sleep staging methods suffer from two major drawbacks. First, there are limited information interactions between modalities in the existing methods. Second, current methods do not develop unified models that can handle different sources of input. To address these issues, we propose a novel sleep stage scoring model sDREAMER, which emphasizes cross-modality interaction and per-channel performance. Specifically, we develop a mixture-of-modality-expert (MoME) model with three pathways for EEG, EMG, and mixed signals with partially shared weights. We further propose a self-distillation training scheme for further information interaction across modalities. Our model is trained with multi-channel inputs and can make classifications on either single-channel or multi-channel inputs. Experiments demonstrate that our model outperforms the existing transformer-based sleep scoring methods for multi-channel inference. For single-channel inference, our model also outperforms the transformer-based models trained with single-channel signals.

sDREAMER: Self-distilled Mixture-of-Modality-Experts Transformer for Automatic Sleep Staging

TL;DR

sDREAMER introduces a self-distilled Mixture-of-Modality-Experts transformer for sleep staging that operates on EEG, EMG, or their combination. By employing three parallel modality pathways with partially shared weights and a cross-modal mix pathway, coupled with a self-distillation scheme, it achieves strong performance in both single-channel and multi-channel inference on a mouse dataset. The architecture integrates patch-based input, epoch- and sequence-level MoME transformers, and a KL-divergence based distillation objective to improve cross-modal interactions. Empirical results show clear advantages over baseline transformer and traditional methods, with ablations confirming the value of joint encoding and self-distillation, and visualizations supporting meaningful, aligned latent representations across modalities.

Abstract

Automatic sleep staging based on electroencephalography (EEG) and electromyography (EMG) signals is an important aspect of sleep-related research. Current sleep staging methods suffer from two major drawbacks. First, there are limited information interactions between modalities in the existing methods. Second, current methods do not develop unified models that can handle different sources of input. To address these issues, we propose a novel sleep stage scoring model sDREAMER, which emphasizes cross-modality interaction and per-channel performance. Specifically, we develop a mixture-of-modality-expert (MoME) model with three pathways for EEG, EMG, and mixed signals with partially shared weights. We further propose a self-distillation training scheme for further information interaction across modalities. Our model is trained with multi-channel inputs and can make classifications on either single-channel or multi-channel inputs. Experiments demonstrate that our model outperforms the existing transformer-based sleep scoring methods for multi-channel inference. For single-channel inference, our model also outperforms the transformer-based models trained with single-channel signals.

Paper Structure

This paper contains 23 sections, 10 equations, 8 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deep-learning-based Sleep Staging
  4. Transformer-based Cross-modality Learning
  5. Self Distillation
  6. Data Collection and Preprocessing
  7. Data Collection
  8. Data Preprocessing
  9. Methodology
  10. Problem Setup
  11. Data Input
  12. Mixture-of-Modality-Experts Module
  13. Epoch Mixture-of-Modality-Experts Transformer
  14. Sequence Mixture-of-Modality-Experts Transformer
  15. Self-Distillation
  16. ...and 8 more sections

Figures (8)

  • Figure 1: The raw EMG and EEG time-series signals.
  • Figure 2: The overall structure of the mixture-of-modality-experts module and the epoch sDREAMER model.
  • Figure 3: Overview of the sequence sDREAMER model. The structure of MoME module has already been illustrated in Fig. \ref{['fig:mome']}.
  • Figure 4: An example of the collected raw data. The two graphs represent the EMG and EEG data, respectively. The background colors of blue, green, and red are the labels of sleep stages.
  • Figure 5: Bar plot of the training set
  • ...and 3 more figures