Garment3DGen: 3D Garment Stylization and Texture Generation

TL;DR

Garment3DGen addresses the challenge of generating simulation-ready 3D garments from a single image by recasting garment creation as a topology-preserving deformation guided by a diffusion-derived pseudo-ground-truth. It integrates 3D-space supervision, 2D differentiable rendering, and garment-specific embeddings (FashionCLIP) with a texture estimation pipeline to produce high-fidelity textured geometries that can be draped onto parametric bodies for physics-based simulation. A diffusion-informed target geometry and a per-triangle Jacobian deformation scheme enable flexible stylization while preserving topology and holes necessary for hand and body interactions. The approach demonstrates strong geometry and texture fidelity across image- and text-guided inputs, enabling practical applications in VR, sketch-to-garment workflows, and rapid asset generation, with public code available.

Abstract

We introduce Garment3DGen a new method to synthesize 3D garment assets from a base mesh given a single input image as guidance. Our proposed approach allows users to generate 3D textured clothes based on both real and synthetic images, such as those generated by text prompts. The generated assets can be directly draped and simulated on human bodies. We leverage the recent progress of image-to-3D diffusion methods to generate 3D garment geometries. However, since these geometries cannot be utilized directly for downstream tasks, we propose to use them as pseudo ground-truth and set up a mesh deformation optimization procedure that deforms a base template mesh to match the generated 3D target. Carefully designed losses allow the base mesh to freely deform towards the desired target, yet preserve mesh quality and topology such that they can be simulated. Finally, we generate high-fidelity texture maps that are globally and locally consistent and faithfully capture the input guidance, allowing us to render the generated 3D assets. With Garment3DGen users can generate the simulation-ready 3D garment of their choice without the need of artist intervention. We present a plethora of quantitative and qualitative comparisons on various assets and demonstrate that Garment3DGen unlocks key applications ranging from sketch-to-simulated garments or interacting with the garments in VR. Code is publicly available.

Figures (13)

• Figure 1: Garment3DGen, automatically transforms a base garment mesh to simulation-ready asset directly from images or text in a frictionless manner unlocking applications such as cloth and hand-cloth interaction in a VR.
• Figure 2: Overview: Given an input 3D base mesh and a target garment image we first generate 3D pseudo ground-truth using a diffusion-based method and utilize the output geometry as a soft supervision signal during the deformation process. Our 3D generated geometry preserves the topology and structure of the base mesh as depicted by the colors of the sleeves/collar while accurately reflecting the geometry and details of the input image. Finally, we introduce a texture-estimation module which outputs the corresponding UV texture that along with the geometry comprise our final generated 3D garment.
• Figure 3: Qualitative Comparisons: We demonstrate several mesh generation methods given an image (top) or text (bottom) and show front and back views of each reconstruction. The 3D Gaussian Splatting 3dgs_triplane method generates distorted frontal colors and dark or blurry back colors while its geometry is not suitable for downstream tasks such as simulation. The second reconstruction approach long2023wonder3dinstant_nsr generates watertight meshes with coarse geometric details and blurry colors. WordRobesrivastava2024wordrobe which is purely text-based, generates simulation-ready garments but they deviate far from the text prompt (e.g., the puffer jacket and t-shirt are not faithful to the prompt). Our proposed approach outputs 3D geometries that are geometrically correct with fine-level texture details that prior works fail to generate.
• Figure 4: Mesh Quality (Top) and Geometry Comparisons (Bottom): We showcase the wireframes of all approaches. Our method stands out as the only one that produces geometries that adhere to the input image while maintaining good mesh quality and incorporating necessary holes for physics-based simulation tasks. At the bottom we showcase the output geometry of various techniques to highlight that our approach captures fine geometric details without geometric artifacts (Wonder3D). Note that several methods produce smooth meshes where details are incorrectly captured in the texture map instead which results in lower quality visual results.
• Figure 5: Applications: Garment3DGen generates textured 3D garments from images, text prompts, simple sketches, that can be fitted to human bodies and drive them with physics-based cloth simulation or even enable interaction between hands and garments in VR.