Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SelfieAvatar: Real-time Head Avatar reenactment from a Selfie Video

Wei Liang, Hui Yu, Derui Ding, Rachael E. Jack, Philippe G. Schyns

TL;DR

This work tackles realistic head avatar reenactment from monocular video by addressing the gap where 3DMM-based methods miss non-facial and background details and GAN-based methods struggle with fine textures. It introduces SelfieAvatar, a real-time pipeline trained from a short selfie video that uses dual StyleGAN generators to synthesize face and non-face regions, guided by a mixed loss to recover high-frequency details. A novel detail texture reconstruction module, including ID-MRF and perceptual losses, enhances edges, wrinkles, and hair textures, producing more authentic head avatars. Experiments on self-reenactment and cross-reenactment with MEAD and IMAvatar show superior reconstruction quality and realism, highlighting practical potential for social signal understanding, gaming, and human-machine interaction, while emphasizing ethical considerations.

Abstract

Head avatar reenactment focuses on creating animatable personal avatars from monocular videos, serving as a foundational element for applications like social signal understanding, gaming, human-machine interaction, and computer vision. Recent advances in 3D Morphable Model (3DMM)-based facial reconstruction methods have achieved remarkable high-fidelity face estimation. However, on the one hand, they struggle to capture the entire head, including non-facial regions and background details in real time, which is an essential aspect for producing realistic, high-fidelity head avatars. On the other hand, recent approaches leveraging generative adversarial networks (GANs) for head avatar generation from videos can achieve high-quality reenactments but encounter limitations in reproducing fine-grained head details, such as wrinkles and hair textures. In addition, existing methods generally rely on a large amount of training data, and rarely focus on using only a simple selfie video to achieve avatar reenactment. To address these challenges, this study introduces a method for detailed head avatar reenactment using a selfie video. The approach combines 3DMMs with a StyleGAN-based generator. A detailed reconstruction model is proposed, incorporating mixed loss functions for foreground reconstruction and avatar image generation during adversarial training to recover high-frequency details. Qualitative and quantitative evaluations on self-reenactment and cross-reenactment tasks demonstrate that the proposed method achieves superior head avatar reconstruction with rich and intricate textures compared to existing approaches.

SelfieAvatar: Real-time Head Avatar reenactment from a Selfie Video

TL;DR

This work tackles realistic head avatar reenactment from monocular video by addressing the gap where 3DMM-based methods miss non-facial and background details and GAN-based methods struggle with fine textures. It introduces SelfieAvatar, a real-time pipeline trained from a short selfie video that uses dual StyleGAN generators to synthesize face and non-face regions, guided by a mixed loss to recover high-frequency details. A novel detail texture reconstruction module, including ID-MRF and perceptual losses, enhances edges, wrinkles, and hair textures, producing more authentic head avatars. Experiments on self-reenactment and cross-reenactment with MEAD and IMAvatar show superior reconstruction quality and realism, highlighting practical potential for social signal understanding, gaming, and human-machine interaction, while emphasizing ethical considerations.

Abstract

Head avatar reenactment focuses on creating animatable personal avatars from monocular videos, serving as a foundational element for applications like social signal understanding, gaming, human-machine interaction, and computer vision. Recent advances in 3D Morphable Model (3DMM)-based facial reconstruction methods have achieved remarkable high-fidelity face estimation. However, on the one hand, they struggle to capture the entire head, including non-facial regions and background details in real time, which is an essential aspect for producing realistic, high-fidelity head avatars. On the other hand, recent approaches leveraging generative adversarial networks (GANs) for head avatar generation from videos can achieve high-quality reenactments but encounter limitations in reproducing fine-grained head details, such as wrinkles and hair textures. In addition, existing methods generally rely on a large amount of training data, and rarely focus on using only a simple selfie video to achieve avatar reenactment. To address these challenges, this study introduces a method for detailed head avatar reenactment using a selfie video. The approach combines 3DMMs with a StyleGAN-based generator. A detailed reconstruction model is proposed, incorporating mixed loss functions for foreground reconstruction and avatar image generation during adversarial training to recover high-frequency details. Qualitative and quantitative evaluations on self-reenactment and cross-reenactment tasks demonstrate that the proposed method achieves superior head avatar reconstruction with rich and intricate textures compared to existing approaches.
Paper Structure (13 sections, 7 equations, 3 figures, 1 table)

This paper contains 13 sections, 7 equations, 3 figures, 1 table.

Table of Contents

  1. INTRODUCTION
  2. Related Works
  3. 3D Face Reconstruction with 3DMM
  4. Head Avatar Creation from a Monocular Video
  5. Method
  6. Overview
  7. Fine Head Avatar Generation
  8. Detail Texture Reconstruction
  9. Experiments
  10. Implementation Details
  11. Results
  12. Conclusion
  13. ETHICAL IMPACT STATEMENT

Figures (3)

  • Figure 1: Overview of the detailed head avatar reconstruction method.
  • Figure 2: Qualitative comparison results on head avatar self-reenactment and cross-reenactment.
  • Figure 3: Visualization results for $\mathcal{M}_{detdail}$. From left to right:(1) ground-truth image of the head avatar, (2) corresponding background, (3) generated head masks with/without $\mathcal{M}_{detdail}$, and (4) output head avatar images.