Remote Blood Oxygen Estimation From Videos Using Neural Networks

TL;DR

This work tackles remote SpO$_2$ estimation from hand videos captured by regular RGB smartphones. It introduces three optophysiology-inspired CNN architectures that operate on RGB skin-color time series extracted from a hand ROI, achieving higher accuracy and explainability than traditional ratio-of-ratios and a prior CNN on both participant-specific and cross-subject splits. The study analyzes the impact of skin type and hand side, and provides RGB-weight visualizations that align with known Hb absorption properties, supporting physiological validity. It also demonstrates transfer to a public contact-based dataset and discusses practical implications for privacy, telehealth, and future motion-robust, shorter-duration measurements.

Abstract

Blood oxygen saturation (SpO$_2$) is an essential indicator of respiratory functionality and is receiving increasing attention during the COVID-19 pandemic. Clinical findings show that it is possible for COVID-19 patients to have significantly low SpO$_2$ before any obvious symptoms. The prevalence of cameras has motivated researchers to investigate methods for monitoring SpO$_2$ using videos. Most prior schemes involving smartphones are contact-based: They require a fingertip to cover the phone's camera and the nearby light source to capture re-emitted light from the illuminated tissue. In this paper, we propose the first convolutional neural network based noncontact SpO$_2$ estimation scheme using smartphone cameras. The scheme analyzes the videos of a participant's hand for physiological sensing, which is convenient and comfortable, and can protect their privacy and allow for keeping face masks on. We design our neural network architectures inspired by the optophysiological models for SpO$_2$ measurement and demonstrate the explainability by visualizing the weights for channel combination. Our proposed models outperform the state-of-the-art model that is designed for contact-based SpO$_2$ measurement, showing the potential of our proposed method to contribute to public health. We also analyze the impact of skin type and the side of a hand on SpO$_2$ estimation performance.

Figures (12)

• Figure 1: The overall pipeline for our proposed method. The hand is first recorded and then the hand and skin pixels are segmented from the background. The average pixel values of the hand for each color channel is calculated frame by frame. This results in an RGB time series which is fed into the neural neural network model for SpO$_2$ estimation.
• Figure 2: Extinction coefficient curves of hemoglobin showing the absorption properties at different wavelengths. The curves were plotted based on ding2018measuringhbcurve. The difference between oxygenated hemoglobin (HbO$_2$) and deoxygenated hemoglobin (Hb) at the red and blue/infrared wavelengths means that these color channels contain useful information for SpO$_2$ prediction by means of optophysiological principles.
• Figure 3: Proposed network structures for predicting an SpO$_2$ level from a fixed-length segment of skin color signals. We highlight the differences among the three model configurations instead of showing the exact model structures. Model 1 combines the RGB channels before temporal feature extraction. Model 2 extracts the temporal features from each channel separately and fuses them toward the end. Model 3 interleaves color channel mixing and temporal feature extraction .
• Figure 4: Fitzpatrick skin types skin-types.
• Figure 5: Setup for capturing hand videos. Over the top is an smartphone device recording a single video for both hands. The left and right hands are in the palm down (PD) and palm up (PU) positions, respectively. The CMS-50E pulse oximeter clamped to the left index finger records reference SpO$_2$ signals.