Facial Kinship Verification from remote photoplethysmography

TL;DR

This study investigates Facial Kinship Verification (FKV) using remote Photoplethysmography (rPPG) signals extracted from facial videos to address privacy and spoofing concerns. The authors propose a Siamese 1D-CNN with a channel-attention module that consumes multi-channel rPPG inputs derived from five traditional rPPG methods, trained with a contrastive loss to produce kinship-discriminative embeddings. Experiments on the UvANEMO Smile Database demonstrate that POS-based rPPG, combined with multi-channel input and channel attention, yields the strongest kinship discrimination on average, establishing a first benchmark for rPPG-based FKV. The work highlights the privacy-preserving and spoof-resilient potential of heart-signal biometrics for kinship verification and points to future work on robustness to noisy data and real-world conditions.

Abstract

Facial Kinship Verification (FKV) aims at automatically determining whether two subjects have a kinship relation based on human faces. It has potential applications in finding missing children and social media analysis. Traditional FKV faces challenges as it is vulnerable to spoof attacks and raises privacy issues. In this paper, we explore for the first time the FKV with vital bio-signals, focusing on remote Photoplethysmography (rPPG). rPPG signals are extracted from facial videos, resulting in a one-dimensional signal that measures the changes in visible light reflection emitted to and detected from the skin caused by the heartbeat. Specifically, in this paper, we employed a straightforward one-dimensional Convolutional Neural Network (1DCNN) with a 1DCNN-Attention module and kinship contrastive loss to learn the kin similarity from rPPGs. The network takes multiple rPPG signals extracted from various facial Regions of Interest (ROIs) as inputs. Additionally, the 1DCNN attention module is designed to learn and capture the discriminative kin features from feature embeddings. Finally, we demonstrate the feasibility of rPPG to detect kinship with the experiment evaluation on the UvANEMO Smile Database from different kin relations.

Figures (5)

• Figure 1: Facial Kinship Verification with Remote Photoplethysmography. We first measure the reflected color changes from facial skin areas resulting in the rPPG signals. Then, the rPPG that includes heartbeat biometric is used as the input to the kinship verification system.
• Figure 2: The general pipeline of the rPPG measurement boccignone2022pyvhrface2ppg. The input is a sequence of facial video frames, with the output signals of rPPG data that implicitly reflect the blood volume changes reflected by the facial skin. Several steps are implemented to generate the rPPG: 1) face detection and alignment, normalizing the cropped faces; 2) ROI selection, choosing the skin regions that contain the PPG-related information; 3) RGB calculation, extracting the raw signal from color spaces; 4) pre-processing, to filter out the noisy data with the frequency filter or standard normalization. 5) rPPG methods, recovering the skin color variations into physiological signals.
• Figure 3: Architecture of the proposed 1D-CNN channel attention model for kinship verification using facial rPPG. The model processes two rPPG inputs sequentially through two 1D convolution layers, a channel attention module, and three fully connected layers. ReLU serves as the activation function, and the Dropout technique is applied at the end to the first two fully connected layers. The network's training is guided by the contrastive loss, which is backpropagated for model updates.
• Figure 4: Data distribution within each kin relation in the subset of UvA-NEMO Smile Database dibekliouglu2012you.
• Figure 5: (a) The ROC curves comparison between different inputs by rPPG methods. (b) The ROC curves comparison for the ablation study. Zoom in for better visualization.