Combining facial videos and biosignals for stress estimation during driving

TL;DR

The paper tackles reliable stress estimation from facial cues in driving, addressing subjective variability and facial masking by using disentangled $3D$ facial geometry from EMOCA. It fuses EMOCA-derived expression/pose features with physiological signals in a Transformer-based temporal model that employs cross-modal attention to compare unimodal, early fusion, and cross-modal strategies. Phase-wise analysis shows that 41 of 56 EMOCA coefficients exhibit consistent, phase-specific stress responses comparable to physiological markers, with velocity-based dynamics being particularly informative. Cross-modal attention fusion of EMOCA and biosignals achieves the best performance (AUROC ~ 0.92, Accuracy ~ 0.866), demonstrating the value of temporal inter-modal interactions for robust stress recognition in driving scenarios and potential safety applications.

Abstract

Reliable stress recognition from facial videos is challenging due to stress's subjective nature and voluntary facial control. While most methods rely on Facial Action Units, the role of disentangled 3D facial geometry remains underexplored. We address this by analyzing stress during distracted driving using EMOCA-derived 3D expression and pose coefficients. Paired hypothesis tests between baseline and stressor phases reveal that 41 of 56 coefficients show consistent, phase-specific stress responses comparable to physiological markers. Building on this, we propose a Transformer-based temporal modeling framework and assess unimodal, early-fusion, and cross-modal attention strategies. Cross-Modal Attention fusion of EMOCA and physiological signals achieves best performance (AUROC 92\%, Accuracy 86.7\%), with EMOCA-gaze fusion also competitive (AUROC 91.8\%). This highlights the effectiveness of temporal modeling and cross-modal attention for stress recognition.

Figures (9)

• Figure 1: EMOCA feature visualization. Left: original infrared video frames with MediaPipe facial landmarks projected on them. Right: FLAME mean-face mesh rendered using the corresponding EMOCA expression and pose coefficients. By projecting the estimated expression and pose parameters onto a fixed mean identity, facial dynamics are disentangled from subject-specific shape.
• Figure 2: Subject-wise low-dimensional embeddings of EMOCA features. From left to right: t-SNE embeddings of expression features for subject T018, UMAP embeddings of pose features for subject T001, UMAP embeddings of expression features for subject T022, and t-SNE embeddings of expression features for subject T044. Blue points denote non-stress and red points denote stress samples. Subject-wise embeddings reveal clear stress-related structure.
• Figure 3: Overview of the proposed visual stress recognition pipeline, where MD video streams are converted into EMOCA-based facial feature sequences augmented with temporal and MD--ND first-order difference cues, followed by Transformer-based temporal modeling and attention pooling for stress classification.
• Figure 4: Cross-modal attention fusion architecture, where EMOCA and biosignal (or gaze-dynamics) streams are independently encoded with convolutional stems and Transformer encoders, fused via bidirectional cross-attention, aggregated using attention pooling, and finally concatenated for stress prediction.
• Figure 5: Phase-wise MD--ND differences highlighting stress-related separability during the stressor phases P2 and P4. Established physiological markers (heart rate, perinasal perspiration) and selected EMOCA facial coefficients (pose_00, exp_40,exp_20,exp_18,exp_03), chosen via PCA for their high correlation with the stress label, exhibit pronounced deviations from baseline, indicating their potential as reliable facial stress trackers.