Passive Heart Rate Monitoring During Smartphone Use in Everyday Life

TL;DR

Passive Heart Rate Monitoring During Smartphone Use in Everyday Life introduces PHRM, a deep-learning system that estimates HR from facial video-based rPPG during ordinary phone use and aggregates readings to daily resting HR (RHR). The method combines an 8-second video HR estimator based on a TS-CNN ensemble with a Kalman-filter–based daily RHR estimator, achieving a $MAPE<10\%$ across skin tones and a $MAE<5$ bpm for RHR in both laboratory and free-living settings. The study uses over $2.0\times 10^5$ videos from hundreds of participants and demonstrates equitable performance across Fitzpatrick/MST skin tone groups, addressing prior biases in rPPG. Daily RHR derived from PHRM correlates with obesity and VO2max, indicating potential for ambient health monitoring and scalable, on-device HR tracking without wearables. Overall, PHRM offers a practical, privacy-conscious pathway to continuous cardiovascular insight from widely available smartphones.

Abstract

Resting heart rate (RHR) is an important biomarker of cardiovascular health and mortality, but tracking it longitudinally generally requires a wearable device, limiting its availability. We present PHRM, a deep learning system for passive heart rate (HR) and RHR measurements during everyday smartphone use, using facial video-based photoplethysmography. Our system was developed using 225,773 videos from 495 participants and validated on 185,970 videos from 205 participants in laboratory and free-living conditions, representing the largest validation study of its kind. Compared to reference electrocardiogram, PHRM achieved a mean absolute percentage error (MAPE) < 10% for HR measurements across three skin tone groups of light, medium and dark pigmentation; MAPE for each skin tone group was non-inferior versus the others. Daily RHR measured by PHRM had a mean absolute error < 5 bpm compared to a wearable HR tracker, and was associated with known risk factors. These results highlight the potential of smartphones to enable passive and equitable heart health monitoring.

Figures (11)

• Figure 1: System overview, development, and validation of the deeplearning system for passive heart rate (HR) and daily resting HR (RHR) measurements (PHRM) during smartphone use. (A) In our research study with consented participants, upon a screen unlock event, the PHRM passively captures, processes, and analyzes 8-sec facial video using a deep neural network (DNN) to estimate HR and associated prediction confidence to determine if the measurement is valid. To compute daily RHR, the PHRM aggregates valid HR measurements from intermittent 8-sec video clips throughout a single day and applies a Kalman filter to improve estimates. (B) Workflow diagram of the different studies used to develop and validate the PHRM system. We used data from five independent, prospective laboratory studies and a prospective free-living study to develop and validate the PHRM.
• Figure 1: Sequence of events in an ambient video recording. Upon screen unlock, the front-facing camera would start at VGA resolution (640x480 px), 15 FPS, and with the phone’s default 3A settings (autoexposure, autofocus, auto-white balance) enabled. If a face was detected within five seconds, the camera would lock the current 3A settings and begin recording frames. Recorded frames were cropped to a stationary bounding box set by the initial position of the face, and saved as Motion JPEG (M-JPEG) at maximum quality to avoid inter-frame compression. The recording would automatically end after 20 seconds had elapsed, if the face moved out of the bounding box, or if the screen was turned off. Clips shorter than 8 seconds were discarded. If the screen was still on, this sequence would restart with the camera reset to its initial settings. This sequence could repeat up to 30 times per screen unlock, for a maximum 10 minutes of video.
• Figure 2: Representative examples of the diversity of free-living data used to validate the PHRM. (A) Illustrative examples of the variety of environments, lighting conditions, front-facing camera angles, and face obstructions for videos captured in the free-living conditions. (B) Examples of facial skin patches randomly sampled from video frames of the cheeks of participants across the full range of Monk skin tones (MST). Videos are sorted by mean brightness across columns and MST across rows (C) From left to right: histograms of the number of 8-sec video clips by the hour of day, illuminance measured by the smartphone ambient light sensor, and the average magnitude of linear acceleration of the smartphone during the videos.
• Figure 2: Accuracy of HR measurements by the PHRM in laboratory settings. (A) Bland Altman plot showing the agreement between PHRM-estimated HR values and the reference ECG measurements. Colors indicate the confidence level of PHRM predictions. Dashed lines show the bias, lower, and upper limits of agreement adjusted for repeated measurements with unequal numbers of replicates. (B) Boxplots showing the distribution of mean absolute percentage error (MAPE) values for individual participants, grouped by skin pigmentation. The box bounds the interquartile range (IQR) divided by the median, and whiskers extend to a maximum of 1.5 × IQR beyond the box. The red dashed line indicates the pre-specified accuracy target of MAPE $<10$%.
• Figure 3: Accuracy of passive HR and RHR measurements by the PHRM in free-living conditions. (A) Bland Altman plot showing the agreement between PHRM-estimated HR values and the reference ECG measurements. Colors indicate the confidence level of PHRM predictions. Dashed lines show the bias, lower, and upper limits of agreement adjusted for repeated measurements with unequal numbers of replicates. (B) Boxplots showing the distribution of mean absolute percentage error (MAPE) values for individual participants, grouped by skin pigmentation. The box bounds the interquartile range (IQR) divided by the median, and whiskers extend to a maximum of 1.5 × IQR beyond the box. The red dashed line indicates the pre-specified accuracy target of MAPE $< 10$%. (C) Bland Altman plot showing the agreement between PHRM-estimated daily RHR values and the reference wearable HR tracker measurements. Colors indicate the day number since the start of RHR predictions. Dashed lines show the bias, lower, and upper limits of agreement adjusted for repeated measurements with unequal numbers of replicates. (D) Mean absolute error (MAE) of PHRM-estimated RHR as a function of day number since the start of RHR predictions, grouped by skin pigmentation. Shaded areas indicate the 95% confidence intervals. The red dashed line indicates the pre-specified accuracy target of MAE $< 5$ bpm.