AvatarCraft: Transforming Text into Neural Human Avatars with Parameterized Shape and Pose Control

TL;DR

AvatarCraft tackles text-driven 3D human avatar creation by guiding neural implicit representations with diffusion models and enabling animation via SMPL-based deformation. The method combines a NeuS-based geometry/color backbone with Instant-NGP for speed, and employs a coarse-to-fine, multi-bbox diffusion-guided optimization along with shape regularization to produce high-fidelity, pose-controllable avatars. The SMPL-guided articulation enables rendering novel views and poses without retraining, and the approach supports realistic scene compositing. Empirical results show improved geometry and texture quality over prior work, with ablations confirming the value of each component and a noted limitation in back-view texture fidelity due to diffusion data distribution.

Abstract

Neural implicit fields are powerful for representing 3D scenes and generating high-quality novel views, but it remains challenging to use such implicit representations for creating a 3D human avatar with a specific identity and artistic style that can be easily animated. Our proposed method, AvatarCraft, addresses this challenge by using diffusion models to guide the learning of geometry and texture for a neural avatar based on a single text prompt. We carefully design the optimization framework of neural implicit fields, including a coarse-to-fine multi-bounding box training strategy, shape regularization, and diffusion-based constraints, to produce high-quality geometry and texture. Additionally, we make the human avatar animatable by deforming the neural implicit field with an explicit warping field that maps the target human mesh to a template human mesh, both represented using parametric human models. This simplifies animation and reshaping of the generated avatar by controlling pose and shape parameters. Extensive experiments on various text descriptions show that AvatarCraft is effective and robust in creating human avatars and rendering novel views, poses, and shapes. Our project page is: https://avatar-craft.github.io/.

Figures (15)

• Figure 1: Method Overview of AvatarCraft. The proposed pipeline is divided into two stages. A) We utilize SDS loss and additional shape regularization to create the template of target avatar using our multiple bounding-box (multi-bbox) and coarse-to-fine (c2f) training strategy. B1) We first use input SMPL parameters to calculate per-vertex rigid transformations. B2.a) Guided by $F_{trg}$, the camera emits rays, and points ${\bm{p}}(t_i)$ on the ray are sampled. B2.b) For all sampled points, we find their corresponding points in the generated canonical space $\mathcal{N}_t$ based on inverse vertex transformation $\mathcal{T}^{-1}$ as well as SMPL mesh $F_{trg}$. B2.c) The color of the rays can be computed using the volumetric rendering equation.
• Figure 2: Coarse-to-Fine and Multi-BBox Training. 1) We partition the canonical space into two bounding boxes for sampling cameras. 2) we use stride ray sampling to render the avatar at different scales. 3) the rendered coarse or fine avatar is interpolated to fit stable diffusion input assumption for calculating the SDS loss.
• Figure 3: Problems with Applying $\nabla\mathcal{L}_{SDS}$ for Avatar Generation. We demonstrate the optimization progress of generation under two conditions: a) without any geometry constraint, and b) with the SDF parameters $\Theta_f$ fixed. Applying no constraint leads to adversarial results, while fixing the SDF parameters results in blurry texture. The prompt for both experiments is "Superman".
• Figure 4: Geometry Generation. AvatarCraft could generate fine geometry detail on the avatar surface. We show the rendered depth map of bare SMPL and "hulk".
• Figure 5: Concept Mixing. AvatarCraft could generate novel avatars by mixing different concepts together.