Efficient and Robust Multidimensional Attention in Remote Physiological Sensing through Target Signal Constrained Factorization

TL;DR

This work targets robustness in remote physiological sensing under domain shifts by introducing TSFM, a Target Signal Constrained Factorization module that guides multidimensional attention with target signal characteristics, and integrating it into MMRPhys, an efficient dual-branch 3D-CNN for simultaneous rPPG and rRSP estimation from RGB and thermal inputs. TSFM leverages a constrained NMF formulation, solved via multiplicative updates, to produce low-rank, temporally smooth embeddings aligned with the physiological target, improving cross-dataset generalization. Across five benchmark datasets, MMRPhys with TSFM achieves state-of-the-art or competitive accuracy for both single-task rPPG and multi-task rPPG/rRSP estimation, while maintaining real-time latency and parameter efficiency. The results highlight the value of signal-guided factorization in attention mechanisms and the benefits of multimodal inputs (RGB and thermal) for robust, real-world vital-sign monitoring in unconstrained environments.

Abstract

Remote physiological sensing using camera-based technologies offers transformative potential for non-invasive vital sign monitoring across healthcare and human-computer interaction domains. Although deep learning approaches have advanced the extraction of physiological signals from video data, existing methods have not been sufficiently assessed for their robustness to domain shifts. These shifts in remote physiological sensing include variations in ambient conditions, camera specifications, head movements, facial poses, and physiological states which often impact real-world performance significantly. Cross-dataset evaluation provides an objective measure to assess generalization capabilities across these domain shifts. We introduce Target Signal Constrained Factorization module (TSFM), a novel multidimensional attention mechanism that explicitly incorporates physiological signal characteristics as factorization constraints, allowing more precise feature extraction. Building on this innovation, we present MMRPhys, an efficient dual-branch 3D-CNN architecture designed for simultaneous multitask estimation of photoplethysmography (rPPG) and respiratory (rRSP) signals from multimodal RGB and thermal video inputs. Through comprehensive cross-dataset evaluation on five benchmark datasets, we demonstrate that MMRPhys with TSFM significantly outperforms state-of-the-art methods in generalization across domain shifts for rPPG and rRSP estimation, while maintaining a minimal inference latency suitable for real-time applications. Our approach establishes new benchmarks for robust multitask and multimodal physiological sensing and offers a computationally efficient framework for practical deployment in unconstrained environments. The web browser-based application featuring on-device real-time inference of MMRPhys model is available at https://physiologicailab.github.io/mmrphys-live

Figures (7)

• Figure 1: Proposed Target Signal Constrained Factorization
• Figure 2: Overview of the proposed TSFM module and MMRPhys architecture. [a]: Deployment of the proposed Target Signal Constrained Factorization module for remote physiological sensing, [b]: MMRPhys: proposed dual-branch 3D-CNN architecture for simultaneous estimation of rPPG and rRSP.
• Figure 3: Comparison of rPPG and rRSP signals estimated on the iBVP Dataset joshi2024ibvp, for MMRPhys trained with TSFM, and FSAM joshi2024factorizephys. The left figures show results from models trained with RGB-Thermal frames from the BP4D+ dataset zhang2016multimodalertugrul2019cross, and the right figures show outputs from models trained only with RGB frames from the SCAMPS dataset mcduff2022scamps.
• Figure 4: Analysis of spatial-temporal features learned by FactorizePhys joshi2024factorizephys with FSAM joshi2024factorizephys and the proposed TSFM. The top row of subplots shows video frames from SCAMPS dataset mcduff2022scamps at different time intervals.
• Figure 5: Analysis of spatial-temporal features learned by MMRPhys using thermal video frames for rRSP estimation. On the left are features for MMRPhys trained with FSAM joshi2024factorizephys, and on the right, features trained with TSFM. The top row in subplots shows video frames from iBVP dataset joshi2024ibvp.