Head360: Learning a Parametric 3D Full-Head for Free-View Synthesis in 360°

TL;DR

Head360 addresses the challenge of free-view 360° head synthesis by building SynHead100 and a hybrid parametric representation that separates shape/motion (52 blendshapes) from appearance (neural texture) and renders with a hex-planes NeRF. It enables hair swapping, single-image fitting, animation, and text-based editing within a single model, and releases public SynHead100 to support research. Experiments show state-of-the-art rendering and editing quality with ablations validating hex-planes and hair detachment. The work provides a publicly released dataset and demonstrates practical pipelines for editable, animatable 360° head synthesis.

Abstract

Creating a 360° parametric model of a human head is a very challenging task. While recent advancements have demonstrated the efficacy of leveraging synthetic data for building such parametric head models, their performance remains inadequate in crucial areas such as expression-driven animation, hairstyle editing, and text-based modifications. In this paper, we build a dataset of artist-designed high-fidelity human heads and propose to create a novel parametric 360° renderable parametric head model from it. Our scheme decouples the facial motion/shape and facial appearance, which are represented by a classic parametric 3D mesh model and an attached neural texture, respectively. We further propose a training method for decompositing hairstyle and facial appearance, allowing free-swapping of the hairstyle. A novel inversion fitting method is presented based on single image input with high generalization and fidelity. To the best of our knowledge, our model is the first parametric 3D full-head that achieves 360° free-view synthesis, image-based fitting, appearance editing, and animation within a single model. Experiments show that facial motions and appearances are well disentangled in the parametric space, leading to SOTA performance in rendering and animating quality. The code and SynHead100 dataset are released at https://nju-3dv.github.io/projects/Head360.

Figures (9)

• Figure 1: Overview. Our model is the first $360^{\circ}$-renderable parametric 3D head with hair that supports image-based fitting and animation simultaneously.
• Figure 2: Overall pipeline. Our model is represented by a neural radiance field with hex-planes, conditioned on a generative neural texture and a parametric 3D mesh model. In this way, the facial appearance, shape, and motion are parameterized as texture code $t$, shape code $s$, and blendshapes parameter $b$, respectively. The RefineNet, a conditional GAN, is introduced to further improve the details of the generated faces.
• Figure 3: Comparison of the data quality.. Our images ( SynHead100 ) exhibit a greater level of detail than Rodin wang2023rodin, including pores, wrinkles, and subtle textures.
• Figure 4: Comparison of fitting results. We compare our method with previous parametric or generative 3D head models in single-image fitting. For a comprehensive comparison, both original models and re-trained models are compared.
• Figure 5: Comparison of generated results.