InceptionHuman: Controllable Prompt-to-NeRF for Photorealistic 3D Human Generation

TL;DR

InceptionHuman addresses the challenge of photorealistic 3D human generation from multi-modal prompts by linking diffusion-based 2D synthesis with Neural Radiance Fields (NeRF) through two progressive modules. Iterative Pose-Aware Refinement (IPAR) and Progressive-Augmented Reconstruction (PAR) enforce cross-view consistency and leverage diffusion priors to produce 3D-consistent, high-fidelity human NeRFs from text, poses, edges, depth, and seed images. The approach achieves state-of-the-art results versus recent text-to-3D baselines, with quantitative gains in semantic consistency and reprojection accuracy, and supports flexible multi-control generation. While primarily focused on static 3D humans, InceptionHuman lays groundwork for future temporally coherent animation by integrating skeletal controls and animation pipelines.

Abstract

This paper presents InceptionHuman, a prompt-to-NeRF framework that allows easy control via a combination of prompts in different modalities (e.g., text, poses, edge, segmentation map, etc) as inputs to generate photorealistic 3D humans. While many works have focused on generating 3D human models, they suffer one or more of the following: lack of distinctive features, unnatural shading/shadows, unnatural poses/clothes, limited views, etc. InceptionHuman achieves consistent 3D human generation within a progressively refined NeRF space with two novel modules, Iterative Pose-Aware Refinement (IPAR) and Progressive-Augmented Reconstruction (PAR). IPAR iteratively refines the diffusion-generated images and synthesizes high-quality 3D-aware views considering the close-pose RGB values. PAR employs a pretrained diffusion prior to augment the generated synthetic views and adds regularization for view-independent appearance. Overall, the synthesis of photorealistic novel views empowers the resulting 3D human NeRF from 360-degree perspectives. Extensive qualitative and quantitative experimental comparison show that our InceptionHuman models achieve state-of-the-art application quality.

Figures (9)

• Figure 1: Prior works on realistic 3D human generation generally suffer from lack of distinctive features eva3d, unnatural clothes and shadows avatarclipcao2023dreamavatartada. Some of them can only synthesize images from limited views eva3dyang20233dhumangan. InceptionHuman is a prompt-driven model for photorealistic 3D human generation that overcomes these limitations.
• Figure 2: InceptionHuman can receive different types of prompts as input and generates a high-quality NeRF-based 3D human. “Any prompts” refer to prompts in many varieties as listed.
• Figure 3: InceptionHuman overview where all RGB images are generated synthetically. Given a combination of prompts (e.g., text prompt an old white man with grey hair wearing blue shirt and optionally others), we first generate a seed image with ControlNet and utilize a mesh estimation module to generate a series of coarse views. a) InceptionHuman introduces Iterative Pose-Aware Refinement (IPAR), where the coarse views are used to reconstruct a radiance field, and we iteratively extract the view-independent components to produce refined views. b) Finally, InceptionHuman adopts the novel Progressive-Augmented Reconstruction (PAR), which includes a convergent and view-independent regularization to match the consistency among the refined images and remove undesirable artifacts.
• Figure 4: Coarse views generated from diffusion models naively suffer from serious 3D inconsistency. This issue can be solved by adopting IPAR and PAR.
• Figure 5: Text-to-3D qualitative comparison. Zoom in for details of InceptionHuman's results and the corresponding depth maps estimated from NeRF. See also the supplemental video.