Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Disentangled Representation for One-shot Progressive Face Swapping

Qi Li, Weining Wang, Chengzhong Xu, Zhenan Sun, Ming-Hsuan Yang

TL;DR

This paper presents a simple yet efficient method, FaceSwapper, for one-shot face swapping based on Generative Adversarial Networks, which achieves state-of-the-art results on benchmark datasets with fewer training samples.

Abstract

Although face swapping has attracted much attention in recent years, it remains a challenging problem. Existing methods leverage a large number of data samples to explore the intrinsic properties of face swapping without considering the semantic information of face images. Moreover, the representation of the identity information tends to be fixed, leading to suboptimal face swapping. In this paper, we present a simple yet efficient method named FaceSwapper, for one-shot face swapping based on Generative Adversarial Networks. Our method consists of a disentangled representation module and a semantic-guided fusion module. The disentangled representation module comprises an attribute encoder and an identity encoder, which aims to achieve the disentanglement of the identity and attribute information. The identity encoder is more flexible, and the attribute encoder contains more attribute details than its competitors. Benefiting from the disentangled representation, FaceSwapper can swap face images progressively. In addition, semantic information is introduced into the semantic-guided fusion module to control the swapped region and model the pose and expression more accurately. Experimental results show that our method achieves state-of-the-art results on benchmark datasets with fewer training samples. Our code is publicly available at https://github.com/liqi-casia/FaceSwapper.

Learning Disentangled Representation for One-shot Progressive Face Swapping

TL;DR

This paper presents a simple yet efficient method, FaceSwapper, for one-shot face swapping based on Generative Adversarial Networks, which achieves state-of-the-art results on benchmark datasets with fewer training samples.

Abstract

Although face swapping has attracted much attention in recent years, it remains a challenging problem. Existing methods leverage a large number of data samples to explore the intrinsic properties of face swapping without considering the semantic information of face images. Moreover, the representation of the identity information tends to be fixed, leading to suboptimal face swapping. In this paper, we present a simple yet efficient method named FaceSwapper, for one-shot face swapping based on Generative Adversarial Networks. Our method consists of a disentangled representation module and a semantic-guided fusion module. The disentangled representation module comprises an attribute encoder and an identity encoder, which aims to achieve the disentanglement of the identity and attribute information. The identity encoder is more flexible, and the attribute encoder contains more attribute details than its competitors. Benefiting from the disentangled representation, FaceSwapper can swap face images progressively. In addition, semantic information is introduced into the semantic-guided fusion module to control the swapped region and model the pose and expression more accurately. Experimental results show that our method achieves state-of-the-art results on benchmark datasets with fewer training samples. Our code is publicly available at https://github.com/liqi-casia/FaceSwapper.
Paper Structure (19 sections, 18 equations, 14 figures, 5 tables)

This paper contains 19 sections, 18 equations, 14 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Disentangled Representation Module
  5. Semantic-guided Fusion Module
  6. Loss Function
  7. Identity and Attribute Preservation Loss
  8. Reconstruction Loss
  9. Adversarial Loss
  10. Total Loss
  11. Relation to Previous Work
  12. Experiments
  13. Datasets and Experimental Settings
  14. Progressive Face Swapping
  15. Experimental Results on the FaceForensics++ Dataset
  16. ...and 4 more sections

Figures (14)

  • Figure 1: Main components of FaceSwapper. It consists of two generators and two discriminators. The generator consists of a disentangled representation module (DRM) and a semantic-guided fusion module (SFM). DRM (blue dashed lines) comprises an attribute encoder and an identity encoder. SFM (green dashed lines) takes the output of the attribute encoder and the output of the identity encoder as inputs and synthesizes an output image that combines the attributes of the former and the identity of the latter. The discriminator is a conditional discriminator trained to distinguish synthetic images from real images while preserving landmark information.
  • Figure 2: Overview of the semantic-guided fusion module. (a) The semantic-guided fusion module integrates the identity and attribute information using multiple semantic-guided face swapping blocks. (b) The specific network structure of the semantic-guided face swapping block, which is built on the semantic-guided denormalization layer. (c) Details of the semantic-guided denormalization layer.
  • Figure 3: Objective functions of our method. (a) The reconstruction loss with respect to different numbers of iterations. (b) The identity loss with respect to different numbers of iterations. (c) The attribute preservation loss with respect to different numbers of iterations. Note that we display only 40,000 iterations for clarity.
  • Figure 4: Progressive face swapping examples with increasing training iterations.
  • Figure 5: Examples of progressive face swapping. The leftmost and second-from-the-left columns represent the source and target images, respectively. The rest columns demonstrate the progressive face swapping results with the increasing coefficients $\lambda _{id}$.
  • ...and 9 more figures