From Sleep Staging to Spindle Detection: Evaluating End-to-End Automated Sleep Analysis

TL;DR

The paper tackles the challenge of end-to-end automated sleep analysis by combining RobustSleepNet for sleep staging and SUMOv2 for spindle detection to replicate expert findings in bipolar disorder versus healthy controls. It demonstrates that automated staging and spindle detection can reach comparable levels of agreement to human experts across multiple datasets, enabling rapid, scalable sleep research. While qualitative replication is strong, some quantitative differences in spindle metrics point to biases and methodological differences with expert annotations, underscoring the need for larger, diverse datasets and standardized benchmarks. The authors publicly release SUMOv2, enhanced code, and SomnoBot to promote privacy-preserving, accessible automation for broad sleep research applications.

Abstract

Automation of sleep analysis, including both macrostructural (sleep stages) and microstructural (e.g., sleep spindles) elements, promises to enable large-scale sleep studies and to reduce variance due to inter-rater incongruencies. While individual steps, such as sleep staging and spindle detection, have been studied separately, the feasibility of automating multi-step sleep analysis remains unclear. Here, we evaluate whether a fully automated analysis using state-of-the-art machine learning models for sleep staging (RobustSleepNet) and subsequent spindle detection (SUMOv2) can replicate findings from an expert-based study of bipolar disorder. The automated analysis qualitatively reproduced key findings from the expert-based study, including significant differences in fast spindle densities between bipolar patients and healthy controls, accomplishing in minutes what previously took months to complete manually. While the results of the automated analysis differed quantitatively from the expert-based study, possibly due to biases between expert raters or between raters and the models, the models individually performed at or above inter-rater agreement for both sleep staging and spindle detection. Our results demonstrate that fully automated approaches have the potential to facilitate large-scale sleep research. We are providing public access to the tools used in our automated analysis by sharing our code and introducing SomnoBot, a privacy-preserving sleep analysis platform.

Figures (4)

• Figure 1: Overview of usual sleep analysis approaches with automated spindle detection in N2 sleep. Applying a spindle detection model to EEG data without accounting for sleep stages (A) is a simple and fully automated approach but may lead to spindles being detected in Wake or REM sleep stages, where they are biologically implausible. More sophisticated approaches involve first annotating sleep stages, either manually (B) or using an automated sleep staging model (C), to restrict spindle detection to N2 (or, additionally, N1 and N3) sleep stages.
• Figure 2: Comparison of model-expert agreement for sleep staging with inter-rater agreement levels among human experts. The average agreement between the RSN model and the BD expert (quantified by mean macro F1 score, x-axis) is shown as a black line, with the shaded region denoting its standard deviation and the red dashed lines representing the individual agreement levels obtained for each subject in the dataset. To compare the model-expert agreement with the agreement between pairs of expert scorers, the histogram (blue bars) and its density estimate (blue curve) show the distribution of macro F1 scores calculated for all subject-wise comparisons of expert pairs in the DODO/H datasets. The blue dotted line shows the average of this distribution. We observe the model-expert agreement level to be comparable to the agreement observed between pairs of human experts.
• Figure 3: Comparison of model-expert agreement for spindle detection with inter-rater agreement among human annotators. The diamond ($\blacklozenge$) and triangle ($\blacktriangle$) symbols indicate the agreement between the SUMOv2 model and the BD expert using expert-annotated N2 sleep stages or N2 sleep stages detected by the RSN model, respectively. We further compared the SUMOv2 model with expert scorers on the DREAMS dataset annotated by two experts ($\bullet$ and $\blacksquare$) and the MODA dataset annotated by a consensus of experts ($+$) or individual experts (average indicated by $\star$, STD as shaded area). To provide context, the histogram (blue bars) and its density estimate (blue curve) show the distribution of agreement levels between pairs of expert scorers on the MODA dataset. The model-expert agreement between SUMOv2 and various experts is comparable to the inter-rater agreement between pairs of experts, with the model-expert agreement slightly decreasing when sleep stages are automatically detected by a model.
• Figure 4: Boxplot of spindle densities for fast spindles (frequencies $>$ 13 Hz) as detected by the SUMOv2 model in the BD recordings for healthy controls (HC) and patients with bipolar disorder (BP). Spindles were detected in N2 sleep annotated by the RSN model. The labels above the boxplots indicate the mean and standard deviation of the spindles per minute (SPM) for each cohort and channel.