UltrAvatar: A Realistic Animatable 3D Avatar Diffusion Model with Authenticity Guided Textures

TL;DR

A novel 3D avatar generation approach termed UltrAvatar with enhanced fidelity of geometry, and superior quality of physically based rendering (PBR)textures without unwanted lighting is proposed, outperforming the state-of-the-art methods by a large margin.

Abstract

Recent advances in 3D avatar generation have gained significant attentions. These breakthroughs aim to produce more realistic animatable avatars, narrowing the gap between virtual and real-world experiences. Most of existing works employ Score Distillation Sampling (SDS) loss, combined with a differentiable renderer and text condition, to guide a diffusion model in generating 3D avatars. However, SDS often generates oversmoothed results with few facial details, thereby lacking the diversity compared with ancestral sampling. On the other hand, other works generate 3D avatar from a single image, where the challenges of unwanted lighting effects, perspective views, and inferior image quality make them difficult to reliably reconstruct the 3D face meshes with the aligned complete textures. In this paper, we propose a novel 3D avatar generation approach termed UltrAvatar with enhanced fidelity of geometry, and superior quality of physically based rendering (PBR) textures without unwanted lighting. To this end, the proposed approach presents a diffuse color extraction model and an authenticity guided texture diffusion model. The former removes the unwanted lighting effects to reveal true diffuse colors so that the generated avatars can be rendered under various lighting conditions. The latter follows two gradient-based guidances for generating PBR textures to render diverse face-identity features and details better aligning with 3D mesh geometry. We demonstrate the effectiveness and robustness of the proposed method, outperforming the state-of-the-art methods by a large margin in the experiments.

Figures (9)

• Figure 1: UltrAvatar. Our method takes a text prompt or a single image as input to generate realistic animatable 3D Avatars with PBR textures, which are compatible with various rendering engines, our generation results in a wide diversity, high quality, and excellent fidelity.
• Figure 2: The Overview of UltrAvatar. First, we feed a text prompt into a generic diffusion model (SDXL podell2023sdxl) to produce a face image. Alternatively, the face image can also be used directly as input into our framework. Second, our DCE model takes the face image to extract its diffuse colors $I_d$ by eliminating lighting. The $I_d$ is then passed to the mesh generator and the edge detector to generate the 3D mesh, camera parameters and the edge image. With these predicted parameters, the initial texture and the corresponding visibility mask can be created by texture mapping. Lastly, we input the masked initial texture into our AGT-DM to generate the PBR textures. A relighting result using the generated mesh and PBR textures is shown here.
• Figure 3: Features Visualization. We render a high-quality data with PBR textures under a complex lighting condition to image $I$, and also render its corresponding ground truth diffuse color image. We input the $I$ to our DCE model to produce result $I_d$. The $S$ is the semantic mask. We apply DDIM inversion and sampling on these images and extract the features. To visualize the features, we apply PCA on the extracted features to check the first three principal components. The attention features and res-features shown here are all from the $8$-th layer at upsampling layers in the U-Net at time step $101$. From the extracted query and key features of $I$, we can clearly visualize the lighting. The colors and extracted query and key features of the result $I_d$ closely match those from the ground truth image, which demonstrates our method effectively removes the lighting. All res-features do not present too much lighting. We also show the color distributions of these three images, illustrating that the result $I_d$ can eliminate shadows and specular points, making its distribution similar to the ground truth.
• Figure 4: DCE Model. The input image $I$ is fed to the face parsing model to create the semantic mask $S$. We apply DDIM inversion on the $I$ and $S$ to get initial noise $z^I_T$ and $z^S_T$, then we progressively denoise the $z^I_T$ and $z^S_T$ to extract and preserve the res-features and attention features separately. Lastly, we progressively denoise the $z^I_T$ one more time, copying the res-features and attention features from storage at certain layers (as discussed in Sec. \ref{['sec:Experiment']}) during sampling to produce ${\hat{z}}_0^I$, the final result $I_d$ will be generated from decoding the ${\hat{z}}_0^I$.
• Figure 5: Results of generating random identities and celebrities. We input the text prompts into the generic SDXL to create 2D face images. Our results showcase the reconstructed high-quality PBR textures which are also well-aligned with the meshes, exhibit high fidelity, and maintain the identity and facial details. To illustrate the quality of our generation, we relight each 3D avatar under various environment maps.