SFDA-rPPG: Source-Free Domain Adaptive Remote Physiological Measurement with Spatio-Temporal Consistency

TL;DR

This work tackles the challenge of cross-domain generalization in remote photoplethysmography (rPPG) under privacy constraints, introducing SFDA-rPPG, a source-free domain adaptation framework. It combines a Three-Branch Spatio-Temporal Consistency Network (TSTC-Net) with a Frequency-domain Wasserstein Distance (FWD) loss to align power spectral density distributions across domains without accessing source data, enabling robust adaptation to unlabeled target domains. The methodology employs a two-stage training regime: first pre-training on the source domain to obtain $H_w=\mathcal{D}_w \circ E_w$ using $\mathcal{L}_{source}$, then adapting with $\mathcal{L}_{adapting}$ across three branches, with inference via the central branch. Extensive cross-dataset experiments on PURE, UBFC-rPPG, and COHFACE, along with ablations, demonstrate that FWD-based spectral alignment and multi-branch consistency significantly improve cross-domain performance and privacy-preserving adaptation; code is provided at the authors' repository.

Abstract

Remote Photoplethysmography (rPPG) is a non-contact method that uses facial video to predict changes in blood volume, enabling physiological metrics measurement. Traditional rPPG models often struggle with poor generalization capacity in unseen domains. Current solutions to this problem is to improve its generalization in the target domain through Domain Generalization (DG) or Domain Adaptation (DA). However, both traditional methods require access to both source domain data and target domain data, which cannot be implemented in scenarios with limited access to source data, and another issue is the privacy of accessing source domain data. In this paper, we propose the first Source-free Domain Adaptation benchmark for rPPG measurement (SFDA-rPPG), which overcomes these limitations by enabling effective domain adaptation without access to source domain data. Our framework incorporates a Three-Branch Spatio-Temporal Consistency Network (TSTC-Net) to enhance feature consistency across domains. Furthermore, we propose a new rPPG distribution alignment loss based on the Frequency-domain Wasserstein Distance (FWD), which leverages optimal transport to align power spectrum distributions across domains effectively and further enforces the alignment of the three branches. Extensive cross-domain experiments and ablation studies demonstrate the effectiveness of our proposed method in source-free domain adaptation settings. Our findings highlight the significant contribution of the proposed FWD loss for distributional alignment, providing a valuable reference for future research and applications. The source code is available at https://github.com/XieYiping66/SFDA-rPPG

Figures (6)

• Figure 1: Overview of cross-domain methods. (a) Multi-source domain generalization lu2023neuron utilizes labeled data from multiple source domains. (b) Unsupervised domain adaptation du2023dual leverages labeled data from both the source and target domains. (c) Source-free domain adaptation utilizes a pretrained model from the source domain and unlabeled data from the target domain.
• Figure 2: The proposed SFDA-rPPG method consists of two distinct stages. Firstly, the pre-training stage involves pre-training the model with labeled source domain data. Secondly, the adaptive stage utilizes a spatio-temporal branching structure for consistency learning, enabling the source model to adapt more effectively to the unlabeled target domain data. The encoder and decoder are disentangled from PhysNet yu2019remote1 to facilitate improved representation learning.
• Figure 3: Distribution alignment comparison diagram of Kullback-Leibler (KL) divergence and Wasserstein distance. $P$ and $Q$ denote two distinct distributions, representing the distribution of two rPPG’s power spectral densities after undergoing softmax transformation. (a) illustrates the vertical probability dependence of KL divergence, where p(x) and q(x) represent the densities in the distributions $P$ and $Q$. And (b) demonstrates the global advantages of the lateral optimal transmission of Wasserstein distance, where $d$ represents the distance to be moved in the adaptation process.
• Figure 4: Generalization evaluation of FWD loss on other backbones.
• Figure 5: Saliency maps of representative samples on PURE dataset for method with or without FWD loss.