Measuring Domain Shifts using Deep Learning Remote Photoplethysmography Model Similarity

TL;DR

This work addresses domain shift in remote photoplethysmography (rPPG) by introducing Centered Kernel Alignment (CKA)-based metrics that quantify activation- and dataset-level similarity across domains. It defines and analyzes three metrics—DS-diff, DS-sim, and Model-sim—with DS-diff and Model-sim correlating well with empirical domain-shift indicators like MAE, while DS-sim is less consistent. Using a 3DCNN-6 rPPG model trained on 21 domains drawn from five datasets, the study demonstrates that DS-diff can be used for model selection even without target ground-truth data, achieving a 13.9% improvement over the average baseline and up to 41.2% over the worst-case baseline. The results underscore the utility of activation-based domain-shift measurements for understanding dataset relationships and for practical deployment decisions in in-the-wild conditions, while also highlighting limitations and avenues for future robustness across architectures.

Abstract

Domain shift differences between training data for deep learning models and the deployment context can result in severe performance issues for models which fail to generalize. We study the domain shift problem under the context of remote photoplethysmography (rPPG), a technique for video-based heart rate inference. We propose metrics based on model similarity which may be used as a measure of domain shift, and we demonstrate high correlation between these metrics and empirical performance. One of the proposed metrics with viable correlations, DS-diff, does not assume access to the ground truth of the target domain, i.e. it may be applied to in-the-wild data. To that end, we investigate a model selection problem in which ground truth results for the evaluation domain is not known, demonstrating a 13.9% performance improvement over the average case baseline.

Figures (6)

• Figure 1: Illustration of a domain shift in feature space. A mild domain shift exists between Domains A and B, and a severe domain shift exists between Domains A and C, and B and C. We propose systems based on model similarity for measuring the domain shifts.
• Figure 2: Sample images from each of the utilized datasets. BP4D+ (Figure \ref{['fig:ds:bp4d']}) and MSPM (Figure \ref{['fig:ds:mspm']}) are further subdivided to generate 21 datasets total.
• Figure 3: Cross dataset evaluation results reporting MAE for models trained on a source dataset and evaluated on a test dataset. MAE values are averaged across the five folds.
• Figure 4: Example CKA map.
• Figure 5: Heatmaps of results for the metrics over every combination of datasets averaged over the five folds. \ref{['fig:domainshift-heatmaps:mae']} shows the MAE as in Figure \ref{['fig:domainshift_xds']}, while the others show their metric for each combination of $ds_x$ and $ds_y$. Figures \ref{['fig:domainshift-heatmaps:DS-sim']} and \ref{['fig:domainshift-heatmaps:Model-sim']} use an inverted colormap for a clear comparison.