My Emotion on your face: The use of Facial Keypoint Detection to preserve Emotions in Latent Space Editing

TL;DR

The paper tackles entanglement in latent-space editing of StyleGAN/2, where altering one attribute often distorts facial expressions. It introduces a Human Face Landmark Detection (HFLD) loss added to an existing latent-space editing framework to constrain facial landmarks and thereby preserve expressions during edits. Empirical results show significant improvements in disentanglement and emotion preservation, including up to a 49% reduction in emotion changes and favorable comparisons against state-of-the-art editors. This approach enables generating diverse face appearances with fixed expressions, enabling efficient data augmentation for facial gesture and expression research while reducing labeling demands.

Abstract

Generative Adversarial Network approaches such as StyleGAN/2 provide two key benefits: the ability to generate photo-realistic face images and possessing a semantically structured latent space from which these images are created. Many approaches have emerged for editing images derived from vectors in the latent space of a pre-trained StyleGAN/2 models by identifying semantically meaningful directions (e.g., gender or age) in the latent space. By moving the vector in a specific direction, the ideal result would only change the target feature while preserving all the other features. Providing an ideal data augmentation approach for gesture research as it could be used to generate numerous image variations whilst keeping the facial expressions intact. However, entanglement issues, where changing one feature inevitably affects other features, impacts the ability to preserve facial expressions. To address this, we propose the use of an addition to the loss function of a Facial Keypoint Detection model to restrict changes to the facial expressions. Building on top of an existing model, adding the proposed Human Face Landmark Detection (HFLD) loss, provided by a pre-trained Facial Keypoint Detection model, to the original loss function. We quantitatively and qualitatively evaluate the existing and our extended model, showing the effectiveness of our approach in addressing the entanglement issue and maintaining the facial expression. Our approach achieves up to 49% reduction in the change of emotion in our experiments. Moreover, we show the benefit of our approach by comparing with state-of-the-art models. By increasing the ability to preserve the facial gesture and expression during facial transformation, we present a way to create human face images with fixed expression but different appearances, making it a reliable data augmentation approach for Facial Gesture and Expression research.

Figures (6)

• Figure 1: We present the variations offered by our approach. The middle image is the original one, all the surrounding images are edited versions. We illustrate that all image have the different appearances but same facial gesture and expression.
• Figure 2: The overview of the training pipeline. The baseline approach (Enjoy your Editing) first sampled the scaler value $s$ for facial features, 40 elements in total. Given a scaler and original code $w$, the trainable matrix $T$ is responsible for generating a editing shift with the degree control by the scalar vector $s$. The $w^\prime$, obtained by $w + Ts$, is further passed through the Generator $G$ to get the edited image. The baseline method contains $\mathcal{L}_{reg}, \mathcal{L}_{D},\mathcal{L}_{content}$ calculated by pre-trained Regressor $R$, Discriminator $D$ and Feature extractor $F$ (Perceptual model-VGG19), respectively. Our proposed loss term $\mathcal{L}_{h}$ is calculated by the pre-trained HFLD model presented in yellow box. All the losses listed in the Figure are used for training of the matrix $T$.
• Figure 3: The prediction result using HFLD model. Given a facial image, the prediction is 106 landmark points.
• Figure 4: The comparison between baseline (w/o HFLD loss) and ours (w/ HFLD loss). Four sets of transformation trajectory are shown, where, in each set, the first row and second row are the results from baseline and our model, respectively. The original image used for transformation is the leftmost image in each set.
• Figure 5: The comparison among AdaTrans, GANSpace and ours. All images in the first row are original images. The second, third and last row show the results from AdaTrans, GANSpace and our model, respectively.