LayerGS: Decomposition and Inpainting of Layered 3D Human Avatars via 2D Gaussian Splatting

TL;DR

LayerGS addresses the challenge of creating animatable multi-layer 3D human avatars by explicitly decomposing clothing from the inner body and enabling realistic virtual try-on from multi-view RGB imagery. It leverages 2D Gaussian Splatting (2DGS) to model layered geometry and employs score-distillation sampling (SDS) with a pretrained diffusion model as a geometry and texture prior to inpaint occluded regions. The method proceeds in three stages: Stage 1 builds a coarse single-layer canonical avatar for outer garment reconstruction, Stage 2 jointly recovers the inner body layer while fixing the outer layer, and Stage 3 refines the outer layer to reduce inter-layer artifacts; meshes are then extracted and Gaussians attached for 3D virtual try-on under novel poses. Evaluations on 4D-Dress and Thuman2.0 show superior rendering quality and layer decomposability compared to state-of-the-art methods, enabling realistic garment swapping and high-fidelity 3D asset creation for immersive applications, with code available at the provided GitHub link.

Abstract

We propose a novel framework for decomposing arbitrarily posed humans into animatable multi-layered 3D human avatars, separating the body and garments. Conventional single-layer reconstruction methods lock clothing to one identity, while prior multi-layer approaches struggle with occluded regions. We overcome both limitations by encoding each layer as a set of 2D Gaussians for accurate geometry and photorealistic rendering, and inpainting hidden regions with a pretrained 2D diffusion model via score-distillation sampling (SDS). Our three-stage training strategy first reconstructs the coarse canonical garment via single-layer reconstruction, followed by multi-layer training to jointly recover the inner-layer body and outer-layer garment details. Experiments on two 3D human benchmark datasets (4D-Dress, Thuman2.0) show that our approach achieves better rendering quality and layer decomposition and recomposition than the previous state-of-the-art, enabling realistic virtual try-on under novel viewpoints and poses, and advancing practical creation of high-fidelity 3D human assets for immersive applications. Our code is available at https://github.com/RockyXu66/LayerGS

Figures (6)

• Figure 1: Given a 3D human scan or multi-view images of a static person, our framework decomposes and inpaints the subject into multiple canonical Gaussian layers for animation and 3D virtual try-on.
• Figure 2: Overview. (Left) input multi-view RGB and segmentation masks. In Stage 1, we learn a single-layer canonical set of 2D Gaussians by integrating with a composition SDS loss and extract a coarse garment mesh by segmentation. In Stage 2, we decompose and optimize an inner Gaussian layer (body and inner garments) while keeping the outer layer fixed. In Stage 3, we refine the outer garment layer with the inner layer frozen. Both resulting Gaussian layers are attached to the mesh to enable high-fidelity 3D virtual try-on under novel poses.
• Figure 3: Qualitative comparison between GALA (with and without 3D scan) and our method (without 3D scan). The top row shows results from GALA trained with a 3D scan, the middle row from GALA trained without a 3D scan, and the bottom row from our approach, which also does not require a 3D scan. From left to right, each column shows: (1) the ground-truth image, (2) recomposed body and garment, (3) posed body, (4) posed garment, (5) canonical body, and (6) canonical garment.
• Figure 4: Qualitative comparison between VTON360 and our method. Top: original subject, front/back garment images, and edited 3DGS renderings from VTON360. Bottom: canonical body, canonical garment, and recomposed results from our method.
• Figure 5: Custom monocular video demo for decompositions.