Not Only Consistency: Enhance Test-Time Adaptation with Spatio-temporal Inconsistency for Remote Physiological Measurement

TL;DR

This work tackles the challenge of deploying remote physiological measurement (RPM) models in unseen environments by introducing a fully Test-Time Adaptation (TTA) framework called Consistency-in-Consistency-integration (CiCi). CiCi leverages two domain priors—the spatio-temporal frequency-domain consistency (STFC) and the time-domain inconsistency (STTI)—in a self-supervised setup, and it uses a Gradient Dynamic Control (GDC) to prevent gradient conflicts during adaptation. The authors implement CiCi with spatio-temporal maps for RGB data (STMap) and mmWave joint time-frequency maps (RJTF), and validate it across five RGB datasets and two mmWave datasets, achieving state-of-the-art performance under fully online adaptation without access to source data. Key findings show that STFC and STTI, when balanced by GDC, provide stable, robust adaptation, yielding improvements in HR, HRV, and mmWave RPM, with notable gains on datasets with strong temporal variability. The approach holds promise for privacy-aware, real-time RPM in diverse real-world settings. CiCi thus advances TTA for RPM by acknowledging and exploiting the inherent time-domain inconsistencies alongside frequency-domain invariances in physiological signals.

Abstract

Remote physiological measurement (RPM) has emerged as a promising non-invasive method for monitoring physiological signals using the non-contact device. Although various domain adaptation and generalization methods were proposed to promote the adaptability of deep-based RPM models in unseen deployment environments, considerations in aspects such as privacy concerns and real-time adaptation restrict their application in real-world deployment. Thus, we aim to propose a novel fully Test-Time Adaptation (TTA) strategy tailored for RPM tasks in this work. Specifically, based on prior knowledge in physiology and our observations, we noticed not only there is spatio-temporal consistency in the frequency domain of BVP signals, but also that inconsistency in the time domain was significant. Given this, by leveraging both consistency and inconsistency priors, we introduce an innovative expert knowledge-based self-supervised \textbf{C}onsistency-\textbf{i}n\textbf{C}onsistency-\textbf{i}ntegration (\textbf{CiCi}) framework to enhances model adaptation during inference. Besides, our approach further incorporates a gradient dynamic control mechanism to mitigate potential conflicts between priors, ensuring stable adaptation across instances. Through extensive experiments on five diverse datasets under the TTA protocol, our method consistently outperforms existing techniques, presenting state-of-the-art performance in real-time self-supervised adaptation without accessing source data. The code will be released later.

Figures (10)

• Figure 1: Illustration of Spatio-temportal Inconsistency in BVP signals. (a) shows a continuous BVP signal from the same facial region, where the power spectral density (PSD) of the two consecutive segments is similar, yet there are differences in signal morphology. (b) demonstrates that within the same timeframe, different facial regions exhibit similar PSDs, but the signal morphology also varies.
• Figure 2: Visualization of Domain Generalization (DG), Domain Adaptation (DA), and Test-Time Adaptation (TTA) methods. Here, $D^S$ represents the source domain data, $D^T$ represents the target domain data, and $Y^S$ represents the label of the source domain data. Notably, unlike DG and DA, TTA does not involve an offline training phase; instead, it performs unsupervised model adaptation during inference.
• Figure 3: The process of generating an STMap from video.
• Figure 4: The process of generating an RJTF map from mmWave signal.
• Figure 5: Illustration of data augmentation on STMaps. In the temporal dimension, it randomly shifts forward by 0-30 frames during sliding segmentation. In the spatial dimension, it randomly scrambles the positions of pixels within the same frame