Efficient Few-shot Identity Preserving Attribute Editing for 3D-aware Deep Generative Models

TL;DR

This work tackles identity-preserving attribute editing in 3D-aware generative models by learning disentangled latent-space edit directions using only few synthetic, mask-based examples. The authors couple a frozen GMPI/StyleGANv2 backbone with an inversion network to estimate per-attribute edits in the $\mathcal{W}^+$ space, applying them via latent offsets and an MPI renderer to achieve 3D-consistent results. They demonstrate that as few as $10$ labeled samples suffice to obtain effective edits, outperforming baselines in FID and ED while closely matching or surpassing them in other metrics, and they validate sequential edits and one-shot stylization for aging. The approach reduces data and computational requirements associated with 3D-aware portrait editing and enables high-resolution, multi-view capable edits without heavy volumetric training. Additionally, pivotal tuning inversion helps preserve high-frequency facial details, broadening practical applicability in real-world scenarios.

Abstract

Identity preserving editing of faces is a generative task that enables modifying the illumination, adding/removing eyeglasses, face aging, editing hairstyles, modifying expression etc., while preserving the identity of the face. Recent progress in 2D generative models have enabled photorealistic editing of faces using simple techniques leveraging the compositionality in GANs. However, identity preserving editing for 3D faces with a given set of attributes is a challenging task as the generative model must reason about view consistency from multiple poses and render a realistic 3D face. Further, 3D portrait editing requires large-scale attribute labelled datasets and presents a trade-off between editability in low-resolution and inflexibility to editing in high resolution. In this work, we aim to alleviate some of the constraints in editing 3D faces by identifying latent space directions that correspond to photorealistic edits. To address this, we present a method that builds on recent advancements in 3D-aware deep generative models and 2D portrait editing techniques to perform efficient few-shot identity preserving attribute editing for 3D-aware generative models. We aim to show from experimental results that using just ten or fewer labelled images of an attribute is sufficient to estimate edit directions in the latent space that correspond to 3D-aware attribute editing. In this work, we leverage an existing face dataset with masks to obtain the synthetic images for few attribute examples required for estimating the edit directions. Further, to demonstrate the linearity of edits, we investigate one-shot stylization by performing sequential editing and use the (2D) Attribute Style Manipulation (ASM) technique to investigate a continuous style manifold for 3D consistent identity preserving face aging. Code and results are available at: https://vishal-vinod.github.io/gmpi-edit/

Figures (7)

• Figure 1: Proposed model architecture. On the left half we have the components of the StyleGANv2 model components (Mapping network, StyleGANv2 generator). In addition we have the Inversion network (depicted in blue at the bottom left). There are up to k pairs of synthetic image pairs created from the CelebA-HQ dataset using the CelebA-HQ Mask dataset's corresponding image masks. The latent difference direction of the inverted image is added to the w-latent of the StyleGANv2 network and passed to the Alpha Map generator (depicted in blue on the left half). Further, the combined results are passed to the MPI (Multi-Plane Image) renderer conditioned on an arbitrary camera pose and finally the rendered output is passed to the pose conditioned discriminator. In the above example we expect the rendered face to also posses the attribute of interest i.e the eyeglasses from the synthetic image pairs (bottom left k image pairs).
• Figure 2: Qualitative multi-view consistent results with GMPI trained on (a) FFHQ dataset, (b) AFHQv2 Cats dataset, (c) MetFaces dataset karras2020training. (Row 1) depicts the RGB images demonstrating camera conditioned MPI rendering, (Row 2) depicts the corresponding depth maps for the RGB images.
• Figure 3: Pictorial demonstration of synthetic image pair creation for inversion. Here (a) represents the ground truth image with the attribute of interest - Eg: eyeglasses, (b) The mask from CelebA-HQ Mask is used to extract the eyeglasses. (c) Negative image i.e any image without the attribute of interest, and (d) Positive image: resulting image from applying cut-and-paste fro the attribute image.
• Figure 4: Experimental results for out-of-distribution image inversion to demonstrate the robustness of the style transfer across datasets using the synthetic image pairs. Here, the first row represents the synthetic image data and the second row shows the latent inversion results. Column (a) represents the input images for identity 3 in the CelebA-HQ dataset. Columns (b), (c), (d) and (e) represents the edits corresponding to eyeglasses, expression, hat and face aging respectively. Notice how the positive images (row 1) appear unrealistic, however, the inversion is in a continuous space demonstrating the GAN's ability to be invariant to inaccuracies when mapping to the latent space demonstrating the underlying hypothesis of the approach making it suitable for multi-view consistency.
• Figure 5: Experimental results for our investigation of inverting out-of-distribution (OOD) images into the method's latent space. In this experiment we use a model trained on the FFHQ dataset and test inversion with images from the CelebA-HQ dataset. In the figure: (a) Ground truth image from the CelebA-HQ dataset, (b) Direct latent inversion with no fine-tuning (c) Pivotal tuning inversion (fine-tuning) in the latent space. Similarly, (d), (e) and (f) shows an image from the CelebA-HQ dataset, direct latent inversion, and pivotal tuning inversion of the image respectively.