Dress-1-to-3: Single Image to Simulation-Ready 3D Outfit with Diffusion Prior and Differentiable Physics

TL;DR

Dress-1-to-3 tackles the problem of turning a casually posed, single-view image into a physics-aware, simulation-ready 3D garment with separable clothing and a posed human. It introduces a unified IPC differentiable framework that optimizes 2D sewing patterns and 3D garment geometry under differentiable simulation (CIPC), guided by multi-view RGB and normal maps produced by a diffusion prior. The method combines a SewFormer-based initial sewing pattern with differentiable pattern optimization, geometric regularizers, and texture generation to deliver realistic, dynamic garments that are ready for physics-based animation. Its evaluation on CloSe and 4D-Dress shows improved geometry reconstruction, accurate sewing-pattern predictions, and convincing textured garment simulations, with ablations confirming the importance of patch symmetrization and the proposed regularizers. The approach enables practical applications in virtual try-on and animation, while highlighting current limitations in multi-layer garment handling, texture fidelity, and reliance on initial sewing-pattern estimates.

Abstract

Recent advances in large models have significantly advanced image-to-3D reconstruction. However, the generated models are often fused into a single piece, limiting their applicability in downstream tasks. This paper focuses on 3D garment generation, a key area for applications like virtual try-on with dynamic garment animations, which require garments to be separable and simulation-ready. We introduce Dress-1-to-3, a novel pipeline that reconstructs physics-plausible, simulation-ready separated garments with sewing patterns and humans from an in-the-wild image. Starting with the image, our approach combines a pre-trained image-to-sewing pattern generation model for creating coarse sewing patterns with a pre-trained multi-view diffusion model to produce multi-view images. The sewing pattern is further refined using a differentiable garment simulator based on the generated multi-view images. Versatile experiments demonstrate that our optimization approach substantially enhances the geometric alignment of the reconstructed 3D garments and humans with the input image. Furthermore, by integrating a texture generation module and a human motion generation module, we produce customized physics-plausible and realistic dynamic garment demonstrations. Project page: https://dress-1-to-3.github.io/

Figures (12)

• Figure 1: Dress-1-to-3 Pipeline. Starting with a single-view input image of a clothed human, we first derive an initial estimation of the sewing pattern. Additionally, we employ multi-view diffusion to generate orbital camera views, which serve as ground-truth 3D information for both human pose and garment shape. Next, we utilize differentiable simulation to sew and drape the pattern onto the posed human model, optimizing its shape and physical parameters in conjunction with geometric regularizers. Finally, the optimized garment shape provides a physically plausible rest shape in its static state and is readily animatable using a physical simulator.
• Figure 2: Sewing Pattern Remeshing. We perform automatic remeshing during optimization when ill-conditioned triangles are detected. To avoid penetration, we pull back the new discretization to the initial unoptimized stage and rerun the garment initialization to fit it onto the human.
• Figure 3: Qualitative Comparisons of Geometry Reconstruction. Our proposed method not only generates sewing patterns that seamlessly integrate into animation and simulation workflows but also achieves superior garment reconstruction accuracy compared to baseline methods.
• Figure 4: Qualitative Comparison of Panel Shape Prediction. Neural Tailor korosteleva2022neuraltailor takes ground-truth garment meshes as input, while SewFormer liu2023towards and our proposed method use single-view images as input. Extra unexpected panels and edges with significant errors are highlighted in red.
• Figure 5: Qualitative Results of Textured Clothed Human. We showcase the generation capability of Dress-1-to-3 using in-the-wild test images from various sources, including both real-world and synthetic images. Our streamlined pipeline generates perfectly fitted 3D garments with visually plausible textures.