StdGEN: Semantic-Decomposed 3D Character Generation from Single Images

TL;DR

StdGEN tackles the challenge of generating decomposed 3D characters from a single image by integrating a Semantic-aware Large Reconstruction Model (S-LRM) with a conditioning multi-view diffusion pipeline and a multi-layer mesh refinement module. The S-LRM jointly encodes geometry, color, and a semantic field to produce differentiable, semantically layered surfaces, enabling body, clothing, and hair separation. Through the Anime3D++ experiments, StdGEN achieves state-of-the-art geometry and texture quality while delivering semantic decomposability at interactive speeds. The work enables practical 2D-to-3D edits and riggable characters for games and film pipelines.

Abstract

We present StdGEN, an innovative pipeline for generating semantically decomposed high-quality 3D characters from single images, enabling broad applications in virtual reality, gaming, and filmmaking, etc. Unlike previous methods which struggle with limited decomposability, unsatisfactory quality, and long optimization times, StdGEN features decomposability, effectiveness and efficiency; i.e., it generates intricately detailed 3D characters with separated semantic components such as the body, clothes, and hair, in three minutes. At the core of StdGEN is our proposed Semantic-aware Large Reconstruction Model (S-LRM), a transformer-based generalizable model that jointly reconstructs geometry, color and semantics from multi-view images in a feed-forward manner. A differentiable multi-layer semantic surface extraction scheme is introduced to acquire meshes from hybrid implicit fields reconstructed by our S-LRM. Additionally, a specialized efficient multi-view diffusion model and an iterative multi-layer surface refinement module are integrated into the pipeline to facilitate high-quality, decomposable 3D character generation. Extensive experiments demonstrate our state-of-the-art performance in 3D anime character generation, surpassing existing baselines by a significant margin in geometry, texture and decomposability. StdGEN offers ready-to-use semantic-decomposed 3D characters and enables flexible customization for a wide range of applications. Project page: https://stdgen.github.io

Figures (19)

• Figure 1: Our StdGEN generates high-quality, decomposed 3D characters from a single reference image.
• Figure 2: The overview of our StdGEN pipeline. Starting from a single reference image, our method utilizes diffusion models to generate multi-view RGB and normal maps, followed by S-LRM to obtain the color/density and semantic field for 3D reconstruction. Semantic decomposition and part-wise refinement are then applied to produce the final result.
• Figure 3: Demonstration of the structure and intermediate outputs of our semantic-aware large reconstruction model (S-LRM).
• Figure 4: Our semantic-equivalent NeRF and SDF extraction scheme (shown in yellow color).
• Figure 5: Qualitative comparisons on geometry and appearance of generated 3D characters.