Facial Spatiotemporal Graphs: Leveraging the 3D Facial Surface for Remote Physiological Measurement

TL;DR

This paper advances remote photoplethysmography by enforcing surface-aligned spatiotemporal modeling of the face. It introduces STGraph, a fixed-topology 3D facial mesh-based representation, and MeshPhys, a lightweight graph-convolutional backbone that processes surface-aware features to estimate the rPPG waveform $\,\hat{Y}$. Through comprehensive intra- and cross-dataset evaluations on four public datasets, the approach achieves state-of-the-art or competitive accuracy with significantly fewer parameters than vision-based baselines, while exhibiting strong generalization due to its explicit 3D-aware inductive bias. Ablation studies confirm the value of surface-aligned node definitions, dense local connectivity, multi-kernel temporal processing, phase-shift robust supervision, and SNR-aware learning, collectively enabling robust, interpretable, and deployable remote physiological measurement.

Abstract

Facial remote photoplethysmography (rPPG) methods estimate physiological signals by modeling subtle color changes on the 3D facial surface over time. However, existing methods fail to explicitly align their receptive fields with the 3D facial surface-the spatial support of the rPPG signal. To address this, we propose the Facial Spatiotemporal Graph (STGraph), a novel representation that encodes facial color and structure using 3D facial mesh sequences-enabling surface-aligned spatiotemporal processing. We introduce MeshPhys, a lightweight spatiotemporal graph convolutional network that operates on the STGraph to estimate physiological signals. Across four benchmark datasets, MeshPhys achieves state-of-the-art or competitive performance in both intra- and cross-dataset settings. Ablation studies show that constraining the model's receptive field to the facial surface acts as a strong structural prior, and that surface-aligned, 3D-aware node features are critical for robustly encoding facial surface color. Together, the STGraph and MeshPhys constitute a novel, principled modeling paradigm for facial rPPG, enabling robust, interpretable, and generalizable estimation. Code is available at https://samcantrill.github.io/facial-stgraph-rppg/ .

Figures (7)

• Figure 1: The Facial STGraph (left) preserves facial structure through an adjacency matrix derived from the facial mesh topology-enforcing local spatiotemporal modeling. STMaps (right) flatten regions into a 2D matrix neglecting relationships between regions—adjacent entries may be distant on the face-limiting localized modeling.
• Figure 2: STGraph construction involves obtaining a fixed-topology 3D mesh $\mathcal{M}_{3D}$ for each frame $I_t$. Leveraging $\mathcal{M}_{3D}$, we compute for each mesh face $f_{t,i}$ the mean RGB value of its projected pixels and use this as the node feature $\mathbf{X}_{t,i}$ for faces that are front-facing with respect to the camera (faces with $\mathbf{d}_{t,i} \cdot \mathbf{d}_{\text{c}} < 0$ are treated as occluded and assigned $\mathbf{X}_{t,i} = \mathbf{0}$). Spatial edges are instantiated between nodes whose corresponding faces share at least one vertex in the canonical UV mesh (plus self-loops), yielding a symmetric, time-invariant adjacency $\mathbf{A}$. Stacking the per-face features $\mathbf{X}_{t,i}$ over time together with the fixed adjacency $\mathbf{A}$ defines the facial spatiotemporal graph $\mathcal{G}_{f} = (\mathcal{N}, \mathcal{E}, \mathbf{X}, \mathbf{A})$.
• Figure 3: MeshPhys operates on the STGraph using a spatiotemporal graph-convolutional network. The MeshPhys backbone extracts and models localized spatiotemporal features encoded on the facial surface through the STGraph and project these features to a waveform estimate through a simple linear prediction head. The backbone consists of five-layers, with each layer comprised of a multi-kernel temporal node-wise convolution block ($MKTCB$), a spatial graph convolution block ($SGCB$), and optionally a spatial graph pooling block ($SGPB$). MeshPhys uses 16 base channels and expands the channel dimension to 32, 64, 128, and 128 in subsequent layers. Spatial pooling is specifically applied in layers 2, 3, and 4, with global spatial pooling performed in the prediction head. Combined, this effectively constrains spatiotemporal modeling to the facial surface.
• Figure 4: Comparison between the estimated and ground-truth pulse rates from intra-dataset evaluation on MMPD. Scatter plot (top) of the estimated versus ground-truth PR, with the identity line indicating perfect agreement. Bland–Altman plot (bottom) showing the difference between the estimated and ground-truth PR as a function of their mean, along with the mean bias and 95% limits of agreement.
• Figure 5: Visualization of the predicted and ground-truth PPG signals in the time domain (top) and frequency domain (bottom) for a representative sample from the test set from intra-dataset evaluation on MMPD. The selected sample corresponds to the median absolute pulse rate error on the test set.