Fully Test-Time rPPG Estimation via Synthetic Signal-Guided Feature Learning

TL;DR

This work tackles the problem of rPPG estimation under domain shifts and unseen heart-rate distributions by introducing fully test-time adaptation (TTA) for rPPG and a synthetic signal-guided feature learning framework. The method pre-trains a base rPPG model, then post-trains a synthetic signal-guided generator to produce latent rPPG features guided by pseudo ground-truth signals, stored in a frequency-based memory bank. During inference, spectral-based entropy minimization and feature-aligned estimation utilize memory-bank data to adapt the model online to unseen domains and HR ranges, with alpha=0.5 and beta=0.1 weighting the alignment terms. A new TTA-rPPG benchmark across COHFACE, PURE, UBFC-rPPG, and VIPL-HR demonstrates superior performance over DG/DA baselines and validates extensive ablations, underscoring the method’s effectiveness for real-world, online rPPG adaptation.

Abstract

Many remote photoplethysmography (rPPG) estimation models have achieved promising performance in the training domain but often fail to accurately estimate physiological signals or heart rates (HR) in the target domains. Domain generalization (DG) or domain adaptation (DA) techniques are therefore adopted during the offline training stage to adapt the model to either unobserved or observed target domains by utilizing all available source domain data. However, in rPPG estimation problems, the adapted model usually encounters challenges in estimating target data with significant domain variation. In contrast, Test-Time Adaptation (TTA) enables the model to adaptively estimate rPPG signals in various unseen domains by online adapting to unlabeled target data without referring to any source data. In this paper, we first establish a new TTA-rPPG benchmark that encompasses various domain information and HR distributions to simulate the challenges encountered in real-world rPPG estimation. Next, we propose a novel synthetic signal-guided rPPG estimation framework to address the forgetting issue during the TTA stage and to enhance the adaptation capability of the pre-trained rPPG model. To this end, we develop a synthetic signal-guided feature learning method by synthesizing pseudo rPPG signals as pseudo ground truths to guide a conditional generator in generating latent rPPG features. In addition, we design an effective spectral-based entropy minimization technique to encourage the rPPG model to learn new target domain information. Both the generated rPPG features and synthesized rPPG signals prevent the rPPG model from overfitting to target data and forgetting previously acquired knowledge, while also broadly covering various heart rate (HR) distributions. Our extensive experiments on the TTA-rPPG benchmark show that the proposed method achieves superior performance.

Figures (9)

• Figure 1: Illustration of different cross-domain scenarios in rPPG estimation: (a) Domain Generalization (DG), (b) Domain Adaptation (DA), and (c) Test-Time Adaptation (TTA). Note that, unlike DG and DA, TTA includes no offline training stage but involves adapting the model in the inference stage, which consists of one online training (adaptation) process and one online testing process.
• Figure 2: The proposed framework consists of one feature extractor $F$, one rPPG estimator $E$, and one latent rPPG feature generator $G$. (a) The pre-training stage: the off-the-shelf rPPG model $T=F\circ E$ is pretrained on the source training data $\{\mathbf{X}_s,\mathbf{S}_s\}$. (b) The post-training stage: the sampled Gaussian noise $\mathbf{n}$ and the synthetic rPPG signal $\widetilde{\mathbf{s}}$ are used to guide the generator $G$ to generate latent rPPG features $\mathbf{z}^G$. (c) The inference stage of test-time adaptation: we first adapt $T$ to the incoming batch of target sample $\{ \mathbf{X}_T\}$ to alleviate the uncertainty of the estimated rPPG signals, and then use the generated paired $\mathbf{z}^G$ and $\widetilde{\mathbf{s}}$ to adapt $T$ to target domain.
• Figure 3: Examples of (a) a ground truth rPPG signal and its corresponding Power Spectral Density (PSD), and (b) a synthetic rPPG signal and its corresponding PSD.
• Figure 4: Structure of the frequency-based memory bank $\mathcal{B}$.
• Figure 5: Examples of the entropy $H(\mathbf{f})$ measured on the PSDs of (a) the estimated rPPG signal $\bar{\mathbf{s}}$, and (b) the ground truth rPPG signal $\mathbf{s}$.