Face Identity Disentanglement via Latent Space Mapping

TL;DR

We address disentangled face representations under minimal supervision by mapping a concatenated identity-and-attribute latent into StyleGAN's latent space $\mathcal{W}$. The method uses two encoders $E_{id}$ and $E_{attr}$, a mapping network $M$, and a pre-trained generator $G$ (StyleGAN) to produce high-fidelity images with identity preserved while varying other facial attributes; a $D_{\mathcal{W}}$ discriminator guides the mapping into $\mathcal{W}$. Empirical results on human heads show strong identity-attribute disentanglement, effective de-identification, and temporally coherent sequences, outperforming class-supervised and reconstruction baselines within StyleGAN's latent-space constraints. Limitations stem from StyleGAN's coverage of the head manifold, suggesting future work on extending to $\mathcal{W}^+$ and applying the framework to other domains.

Abstract

Learning disentangled representations of data is a fundamental problem in artificial intelligence. Specifically, disentangled latent representations allow generative models to control and compose the disentangled factors in the synthesis process. Current methods, however, require extensive supervision and training, or instead, noticeably compromise quality. In this paper, we present a method that learns how to represent data in a disentangled way, with minimal supervision, manifested solely using available pre-trained networks. Our key insight is to decouple the processes of disentanglement and synthesis, by employing a leading pre-trained unconditional image generator, such as StyleGAN. By learning to map into its latent space, we leverage both its state-of-the-art quality, and its rich and expressive latent space, without the burden of training it. We demonstrate our approach on the complex and high dimensional domain of human heads. We evaluate our method qualitatively and quantitatively, and exhibit its success with de-identification operations and with temporal identity coherency in image sequences. Through extensive experimentation, we show that our method successfully disentangles identity from other facial attributes, surpassing existing methods, even though they require more training and supervision.

Figures (21)

• Figure 1: Our disentanglement framework uses two encoders (left) to generate the latent code $z$, consisting of a description of the property of interest ($f$), and all the rest ($\overline{f}$). The code is then mapped to the latent space $\mathcal{W}$ of the employed pre-trained generator $G$. This decouples the tasks of learning quality image generation and disentanglement.
• Figure 2: Sample results generated by our method, demonstrating the ability to disentangle identity from other facial attributes: pose, expression and illumination and preserving one while manipulating the other. Three images are used as input, forming a 3 by 3 table combinations generated by our method. As can be seen, identity is preserved along the columns, and attributes are preserved along the rows.
• Figure 3: Our disentanglement scheme, as utilized in the human head domain. Data flow is marked by solid lines, and losses by dashed ones. The identity and attributes codes are first extracted from two input images using encoders $E_{id}$ and $E_{attr}$ respectively. Through our mapping network $M$, the concatenated codes are mapped to $\mathcal{W}$, the latent space of the pre-trained generator $G$, which in turn generates the resulting image. An adversarial loss $\mathcal{L}_{adv}$ ensures proper mapping to the $\mathcal{W}$ space. Identity preservation is encouraged using $\mathcal{L}_{id}$, that penalizes differences in identity between $I_{id}, I_{out}$. Attributes preservation is encouraged using $\mathcal{L}_{rec}$, $\mathcal{L}_{lnd}$, that penalizes pixel-level and facial landmarks differences respectively, between $I_{attr}, I_{out}$.
• Figure 4: Feature combination results. For every image in the table, identity is taken from the top, and the rest of the attributes (including expressions, orientation, lighting conditions, etc.) from the left. All images (both inputs and output) were generated using StyleGAN.
• Figure 5: Feature combination results on FFHQ images. The setting is identical to \ref{['fig:table_results_1']}.