Semi-rPPG: Semi-Supervised Remote Physiological Measurement with Curriculum Pseudo-Labeling

TL;DR

A novel semi-supervised learning (SSL) method named semi-rPPG that combines curriculum pseudo-labeling and consistency regularization is proposed to extract intrinsic physiological features from unlabeled data without impairing the model from noises.

Abstract

Remote Photoplethysmography (rPPG) is a promising technique to monitor physiological signals such as heart rate from facial videos. However, the labeled facial videos in this research are challenging to collect. Current rPPG research is mainly based on several small public datasets collected in simple environments, which limits the generalization and scale of the AI models. Semi-supervised methods that leverage a small amount of labeled data and abundant unlabeled data can fill this gap for rPPG learning. In this study, a novel semi-supervised learning method named Semi-rPPG that combines curriculum pseudo-labeling and consistency regularization is proposed to extract intrinsic physiological features from unlabelled data without impairing the model from noises. Specifically, a curriculum pseudo-labeling strategy with signal-to-noise ratio (SNR) criteria is proposed to annotate the unlabelled data while adaptively filtering out the low-quality unlabelled data. Besides, a novel consistency regularization term for quasi-periodic signals is proposed through weak and strong augmented clips. To benefit the research on semi-supervised rPPG measurement, we establish a novel semi-supervised benchmark for rPPG learning through intra-dataset and cross-dataset evaluation on four public datasets. The proposed Semi-rPPG method achieves the best results compared with three classical semi-supervised methods under different protocols. Ablation studies are conducted to prove the effectiveness of the proposed methods.

Figures (8)

• Figure 1: Comparison of deep learning methods for rPPG learning: (a) Supervised methods: utilize labeled data for training; (b) Self-supervised methods: leverage unlabeled data for training; (c) Semi-supervised methods: use a small amount of labeled data and a large amount of unlabeled data for training
• Figure 2: The overall framework of the proposed Semi-rPPG. (a) In the supervised stage, the model is trained on labeled data using supervised loss and consistency regularization. (b) During the pseudo-labeling stage, the trained model assigns preliminary pseudo-labels to the unlabeled data. Next, the signal-to-noise ratio (SNR) is computed and sorted for all unlabeled samples. Top high-quality pseudo-labels are then adaptively chosen based on a curriculum ratio $R$. These selected pseudo-labels, along with the corresponding unlabeled data, are combined with the labeled dataset and used for iterative supervised training, as illustrated in section (a). (c) The facial clip is weakly and strongly augmented with a small temporal shift and a temporal reverse, respectively. A consistency regularization term is proposed as the heart rate shall remain consistent across these two augmentations.
• Figure 3: An illustration of calculating heart rate from blood volume pulse signal (BVP): First, the BVP signal is converted into power spectral density (PSD) through Fast Fourier Transform; Next, the PSD is categorized into classes ranging from 0 to 140, which correspond to heart rates of 40 to 180 beat per minute (BPM); Finally, the heart rate is determined by reverse mapping the class.
• Figure 4: Illustration of training datasets: the four datasets have different races, skin tones, illumination conditions, backgrounds, and recording devices. There are domain gaps among various datasets. The numbers of subjects for these datasets are 107, 40, 10, and 42. The colored videos for each dataset are 2378, 160, 60, and 42, respectively. The subject's number and video number are relatively small, especially for the two most commonly used datasets of PURE and UBFC.
• Figure 5: The variations of SNR and IPR across training epochs