Design2Cloth: 3D Cloth Generation from 2D Masks

TL;DR

Design2Cloth addresses the challenge of realistic 3D garment generation by learning from a large real-world dataset and enabling user-friendly garment design through 2D visibility masks. It introduces a differentiable implicit garment GAN with a tri-plane generator that produces an unsigned distance field $d(\mathbf{p})$, decoded by MeshUDF, conditioned on latent codes $\mathbf{z}$ derived from a mask encoder and SMPL shape parameters. A dual-resolution discriminator enforces both global structure and high-frequency wrinkles, yielding garments with realistic creases and textures. Trained on DigitalMe, consisting of 2010 subjects and over 2k garments, the model achieves state-of-the-art reconstruction and generation, supports interpolation across styles and shapes, and enables high-quality 3D garment reconstruction from in-the-wild images and scans. The approach promises practical impact for digital fashion, avatar realism, and plug-and-play 3D garment reconstruction workflows, with publicly available data, code, and models.

Abstract

In recent years, there has been a significant shift in the field of digital avatar research, towards modeling, animating and reconstructing clothed human representations, as a key step towards creating realistic avatars. However, current 3D cloth generation methods are garment specific or trained completely on synthetic data, hence lacking fine details and realism. In this work, we make a step towards automatic realistic garment design and propose Design2Cloth, a high fidelity 3D generative model trained on a real world dataset from more than 2000 subject scans. To provide vital contribution to the fashion industry, we developed a user-friendly adversarial model capable of generating diverse and detailed clothes simply by drawing a 2D cloth mask. Under a series of both qualitative and quantitative experiments, we showcase that Design2Cloth outperforms current state-of-the-art cloth generative models by a large margin. In addition to the generative properties of our network, we showcase that the proposed method can be used to achieve high quality reconstructions from single in-the-wild images and 3D scans. Dataset, code and pre-trained model will become publicly available.

Figures (8)

• Figure 1: We proposed Design2Cloth: a high fidelity 3D generative model for garment generation from simple 2D masks, trained on a large scale 3D cloth dataset of real-world scans. Figure illustrates interpolation between shirt (Left), trousers (Right) and shapes.
• Figure 2: Overview of the proposed Design2Cloth: A binary mask $\mathbf{M}$ along with a shape vector $\boldsymbol{\beta}$ are fed to the encoder modules, $\textbf{E}_m, \textbf{E}_{\beta}$ to produce a latent vector $\mathbf{z}$ that is used to drive the triplane generator $\mathbf{G}_t$. The decoder network takes as input the triplane features of the projected points and regresses their corresponding unsigned distance function that is then meshed using the differentiable MeshUDF guillard2022meshudf. To enforce the generation of highly detailed clothes we utilize a dual resolution discriminator network $\mathcal{D}$, that take as input two sparse point clouds sampled from the surface of the generated cloth. The low frequency branch $\textbf{D}_l$ takes as input a uniformly sampled point cloud whereas the high frequency branch $\textbf{D}_h$ takes as input a point cloud sampled from coarse areas of the garment surface.
• Figure 3: Qualitative comparison between the proposed and DrapeNet methods of reconstruction performance on DigitalMe (Top), Cloth3D bertiche2020cloth3d and ClothesNet zhou2023clothesnet (Bottom) datasets.
• Figure 4: Interpolation between source and target styles (Top) and shapes (Bottom). The proposed method can interpolate between shapes and styles, generating realistic intermediate clothes.
• Figure 5: Human evaluation results. Left: Average realism scores of the generated and the ground truth data. Right: Perceptual Evaluation of 3D reconstructions from in-the-wild images between the proposed and the baseline methods.