HeadSculpt: Crafting 3D Head Avatars with Text

TL;DR

HeadSculpt tackles the challenge of text-driven 3D head avatar generation and editing by injecting 3D priors into a pre-trained 2D diffusion framework. It introduces a prior-driven score distillation pipeline that uses landmark-based ControlNet and a learned back-view token, along with Identity-aware editing score distillation to preserve identity during edits. The method adopts a coarse-to-fine architecture (NeRF for geometry, DMTet for high-resolution texture) and demonstrates superior fidelity and fine-grained editing compared with state-of-the-art baselines. Results show improved view-consistency, reduced Janus artifacts, and robust editing across a wide range of prompts. This approach advances practical text-to-3D head generation for applications in AR/VR and gaming while highlighting considerations for responsible deployment.

Abstract

Recently, text-guided 3D generative methods have made remarkable advancements in producing high-quality textures and geometry, capitalizing on the proliferation of large vision-language and image diffusion models. However, existing methods still struggle to create high-fidelity 3D head avatars in two aspects: (1) They rely mostly on a pre-trained text-to-image diffusion model whilst missing the necessary 3D awareness and head priors. This makes them prone to inconsistency and geometric distortions in the generated avatars. (2) They fall short in fine-grained editing. This is primarily due to the inherited limitations from the pre-trained 2D image diffusion models, which become more pronounced when it comes to 3D head avatars. In this work, we address these challenges by introducing a versatile coarse-to-fine pipeline dubbed HeadSculpt for crafting (i.e., generating and editing) 3D head avatars from textual prompts. Specifically, we first equip the diffusion model with 3D awareness by leveraging landmark-based control and a learned textual embedding representing the back view appearance of heads, enabling 3D-consistent head avatar generations. We further propose a novel identity-aware editing score distillation strategy to optimize a textured mesh with a high-resolution differentiable rendering technique. This enables identity preservation while following the editing instruction. We showcase HeadSculpt's superior fidelity and editing capabilities through comprehensive experiments and comparisons with existing methods.

Figures (17)

• Figure 1: Examples of generation and editing results obtained using the proposed HeadSculpt. It enables the creation and fine-grained editing of high-quality head avatars, featuring intricate geometry and texture, for any type of head avatar using simple descriptions or instructions. Symbols indicate the following prompt prefixes: $\ast$ "a head of [text]" and $\dagger$ "a DSLR portrait of [text]". HTML]EFEFEFThe captions in gray are the prompt suffixes while HTML]DAE8FCthe blue ones are the editing instructions.
• Figure 2: Overall architecture of HeadSculpt. We craft high-resolution 3D head avatars in a coarse-to-fine manner. (a) We optimize neural field representations for the coarse model. (b) We refine or edit the model using the extracted 3D mesh and apply identity-aware editing score distillation if editing is the target. (c) The core of our pipeline is the prior-driven score distillation, which incorporates landmark control, enhanced view-dependent prompts, and an InstructPix2Pix branch.
• Figure 3: Generation results with various shapes. The first row shows three randomly sampled FLAME models, while the second row presents our generated results (incl. normals) using these FLAME models as initialization. All results are under the same text prompt.
• Figure 4: More specific editing results.$\ddagger$ Instruction prefix: make his expression as[text].
• Figure 5: Impact of the edit scale $\omega_e$ in IESD. It balances the preservation of the initial appearance and the extent of the desired editing, making the editing process more controllable and flexible.