HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation

TL;DR

HumanNorm tackles the gap in text-to-3D human generation by learning a normal diffusion model that augments 2D diffusion priors with geometry-aware perception. It introduces a normal-adapted and depth-adapted diffusion framework for geometry and a normal-aligned diffusion model for texture, coupled with progressive geometry generation and multi-step SDS losses to reduce artifacts. The method yields high-fidelity geometry with wrinkles and realistic textures, outperforming state-of-the-art text-to-3D methods in both geometry and texture quality, as shown by qualitative, quantitative, and user studies. The approach enables exporting riggable meshes and texture maps and offers editing and animation capabilities, advancing practical deployment in AR/VR and metaverse contexts.

Abstract

Recent text-to-3D methods employing diffusion models have made significant advancements in 3D human generation. However, these approaches face challenges due to the limitations of text-to-image diffusion models, which lack an understanding of 3D structures. Consequently, these methods struggle to achieve high-quality human generation, resulting in smooth geometry and cartoon-like appearances. In this paper, we propose HumanNorm, a novel approach for high-quality and realistic 3D human generation. The main idea is to enhance the model's 2D perception of 3D geometry by learning a normal-adapted diffusion model and a normal-aligned diffusion model. The normal-adapted diffusion model can generate high-fidelity normal maps corresponding to user prompts with view-dependent and body-aware text. The normal-aligned diffusion model learns to generate color images aligned with the normal maps, thereby transforming physical geometry details into realistic appearance. Leveraging the proposed normal diffusion model, we devise a progressive geometry generation strategy and a multi-step Score Distillation Sampling (SDS) loss to enhance the performance of 3D human generation. Comprehensive experiments substantiate HumanNorm's ability to generate 3D humans with intricate geometry and realistic appearances. HumanNorm outperforms existing text-to-3D methods in both geometry and texture quality. The project page of HumanNorm is https://humannorm.github.io/.

Figures (14)

• Figure 1: 2D results by normal-adapted and depth-adapted diffusion models. The view-dependent texts like "front view" are utilized to control the view direction. The body-aware texts like "upper body" are employed to control which body part is generated.
• Figure 2: Problems of existing methods.
• Figure 3: Overview of HumanNorm. Our method is designed for high-quality and realistic 3D human generation from given prompts. The whole framework consists of geometry and texture generation. We first propose the normal-adapted and depth-adapted diffusion model for the geometry generation. These two models can guide the rendered normal and depth maps to approach the learned distribution of high-fidelity normal and depth maps through the SDS loss, thereby achieving high-quality geometry generation. In terms of texture generation, we introduce the normal-aligned diffusion model. The normal-aligned diffusion model leverages normal maps as guiding cues to ensure the alignment of the generated texture with geometry. We first exclusively employ the SDS loss and then incorporate the multi-step SDS and perceptual loss to achieve realistic texture generation.
• Figure 4: Examples of 3D humans generated by HumanNorm. A single view and the corresponding normal map are rendered for visualization. See supplementary for video results.
• Figure 5: Comparisons with text-to-3D content methods and text-to-3D human methods. The results of DreamFusion are generated by unofficial code. The results of DreamHuman are taken from its original paper and project page.