Analyzing Participants' Engagement during Online Meetings Using Unsupervised Remote Photoplethysmography with Behavioral Features

TL;DR

This work tackles measuring engagement in online meetings without physical sensors by extracting heart rate variability (HRV) signals from video using unsupervised remote photoplethysmography (rPPG) and fusing them with behavioral cues. Leveraging a novel Engagement Dataset collected in real-world group meetings, the authors reconstruct rPPG with Contrast-Phys, derive HRV features, and perform a two-stage engagement classification with feature-level fusion of HRV and facial/motion cues. Key findings show that HRV features alone can achieve high accuracy, reaching around 0.94 with 2–4 minute observation windows, and that incorporating behavioral features can boost accuracy to about 0.96, with BF4 and BF5 providing the strongest gains. The approach demonstrates practical, non-intrusive engagement estimation suitable for real-time monitoring and group-dynamics analysis, with potential applications in healthcare, education, and workplace stress assessment.

Abstract

Engagement measurement finds application in healthcare, education, services. The use of physiological and behavioral features is viable, but the impracticality of traditional physiological measurement arises due to the need for contact sensors. We demonstrate the feasibility of unsupervised remote photoplethysmography (rPPG) as an alternative for contact sensors in deriving heart rate variability (HRV) features, then fusing these with behavioral features to measure engagement in online group meetings. Firstly, a unique Engagement Dataset of online interactions among social workers is collected with granular engagement labels, offering insight into virtual meeting dynamics. Secondly, a pre-trained rPPG model is customized to reconstruct rPPG signals from video meetings in an unsupervised manner, enabling the calculation of HRV features. Thirdly, the feasibility of estimating engagement from HRV features using short observation windows, with a notable enhancement when using longer observation windows of two to four minutes, is demonstrated. Fourthly, the effectiveness of behavioral cues is evaluated when fused with physiological data, which further enhances engagement estimation performance. An accuracy of 94% is achieved when only HRV features are used, eliminating the need for contact sensors or ground truth signals; use of behavioral cues raises the accuracy to 96%. Facial analysis offers precise engagement measurement, beneficial for future applications.

Figures (6)

• Figure 1: A large Engagement Dataset of realistic online meetings. Facial videos and cPPG were recorded.
• Figure 2: The diagram depicts engagement estimation based on heart rate variability (HRV) and behavioral features. HRV features are calculated from reconstructed rPPG signals, and behavioral features are computed using the OpenFace library baltrusaitis2018openface. Feature-level multimodality fusion is employed, followed by a classifier.
• Figure 3: Observation windows concept for extracting BF and HRV features in engagement analysis.
• Figure 4: Ensemble of knn and Random Forest model's confusion matrices for engagement estimation (HRV observation window of 240 seconds) based on (a) HRV features; (b) HRV and Behavioral features.
• Figure 5: Accuracy of the ensemble of knn and Random Forest model for engagement estimation based on HRV and both (4) and (5) BF calculated at different values of HRV observation window. The accuracy of the model, based solely on BF, remains constant due to its independence from the value of the HRV window.