EventSleep: Sleep Activity Recognition with Event Cameras

TL;DR

EventSleep introduces a first-of-its-kind dataset and processing pipeline for sleep activity recognition under dark conditions using event cameras. The authors combine online event frame representations with optional grayscale reconstruction, evaluate two backbones (Dinov2-ER and ResNet-E), and integrate Bayesian uncertainty via Laplace ensembles to obtain calibrated predictions suitable for medical use. Experimental results show strong performance of event-based methods, with Laplace ensembles delivering superior calibration and robustness to noise, while online framing highlights opportunities and challenges for real-time sleep monitoring. Overall, this work demonstrates the potential of event cameras for non-invasive, privacy-preserving sleep evaluation and lays groundwork for uncertainty-aware, online sleep analysis tools.

Abstract

Event cameras are a promising technology for activity recognition in dark environments due to their unique properties. However, real event camera datasets under low-lighting conditions are still scarce, which also limits the number of approaches to solve these kind of problems, hindering the potential of this technology in many applications. We present EventSleep, a new dataset and methodology to address this gap and study the suitability of event cameras for a very relevant medical application: sleep monitoring for sleep disorders analysis. The dataset contains synchronized event and infrared recordings emulating common movements that happen during the sleep, resulting in a new challenging and unique dataset for activity recognition in dark environments. Our novel pipeline is able to achieve high accuracy under these challenging conditions and incorporates a Bayesian approach (Laplace ensembles) to increase the robustness in the predictions, which is fundamental for medical applications. Our work is the first application of Bayesian neural networks for event cameras, the first use of Laplace ensembles in a realistic problem, and also demonstrates for the first time the potential of event cameras in a new application domain: to enhance current sleep evaluation procedures. Our activity recognition results highlight the potential of event cameras under dark conditions, and its capacity and robustness for sleep activity recognition, and open problems as the adaptation of event data pre-processing techniques to dark environments.

Figures (9)

• Figure 1: Overview of EventSleep. The left part summarizes the dataset, showing the room used to record, the three configurations considered, depending on the blanket and bedlamp state, and the ten sleep activity classes labeled. The right part shows the stages of the pipeline presented. Our approach for activity recognition from event data includes a Bayesian approach that achieves the best performance and robustness to noise and misclassification errors.
• Figure 2: EventSleep classes. Each plot shows an event frame (red and green pixels) of each class superposed to the corresponding infrared frame (grayscale).
• Figure 3: Privacy comparison: IR frame (left) vs reconstruction from events Rebecq19pami (right). More examples in Sec. \ref{['Sec:SuppConfig']}.
• Figure 4: ResNet-E online qualitative results in full sequences. Each bar corresponds to a complete trial and is colored according to the clip ground truth (GT) or the frame model prediction (Pred).
• Figure 5: Three examples: (1) In-distribution frame where all the models predict correctly. (2) All the models misclassify the static pose frame but ensemble methods identify the uncertainty among static labels. (3) Almost empty frame where Laplace approximation reduces confidence.