GarmageNet: A Multimodal Generative Framework for Sewing Pattern Design and Generic Garment Modeling

TL;DR

GarmageNet presents a unified pipeline to translate multimodal design inputs into production-ready digital garments by introducing Garmage, a panel-aligned geometry-image representation that pairs 2D sewing patterns with 3D drape. A latent-diffusion transformer generates complete Garmages, while GarmageJigsaw recovers vertex-level sewing relationships to produce vectorized sewing patterns and simulation-ready meshes. The authors also introduce GarmageSet, a large-scale, professionally annotated dataset that enables robust, real-world training and evaluation. Across multiple tasks, GarmageNet demonstrates superior fidelity, robustness, and versatility for multimodal garment generation, automatic modeling from patterns, seam recovery, and progressive editing, potentially accelerating digital fashion pipelines and manufacturing workflows.

Abstract

Realistic digital garment modeling remains a labor-intensive task due to the intricate process of translating 2D sewing patterns into high-fidelity, simulation-ready 3D garments. We introduce GarmageNet, a unified generative framework that automates the creation of 2D sewing patterns, the construction of sewing relationships, and the synthesis of 3D garment initializations compatible with physics-based simulation. Central to our approach is Garmage, a novel garment representation that encodes each panel as a structured geometry image, effectively bridging the semantic and geometric gap between 2D structural patterns and 3D garment geometries. Followed by GarmageNet, a latent diffusion transformer to synthesize panel-wise geometry images and GarmageJigsaw, a neural module for predicting point-to-point sewing connections along panel contours. To support training and evaluation, we build GarmageSet, a large-scale dataset comprising 14,801 professionally designed garments with detailed structural and style annotations. Our method demonstrates versatility and efficacy across multiple application scenarios, including scalable garment generation from multi-modal design concepts (text prompts, sketches, photographs), automatic modeling from raw flat sewing patterns, pattern recovery from unstructured point clouds, and progressive garment editing using conventional instructions, laying the foundation for fully automated, production-ready pipelines in digital fashion. Project page: https://style3d.github.io/garmagenet/.

Figures (19)

• Figure 1: Trade-offs between structure-centric and geometry-centric approaches. The magic of wearing (a) indicates that the same sewing pattern could lead to various draping statuses, raising the problem of Structure-centric modeling, which focuses on sewing pattern structure but fails to ensure draping alignment with the input (b). On the other hand, Geometry-centric modeling focuses on draping alignment but fails to preserve structure integrity in UV space (c).
• Figure 2: Overview of our GarmageNet framework, which seamlessly converts multi-modal design inputs—including text descriptions, sewing patterns, line-art sketches, and point clouds (a)—into simulation-ready garment assets (d). Central to our framework is the novel Garmage representation, a 2D–3D unified representation that encodes each garment as a structured set of per-panel geometry images (b). Leveraging Garmage, our approach efficiently recovers vertex-level sewing relationships and detailed 3D draping initializations (c), enabling direct and high-quality garment simulation.
• Figure 3: Overview of our GarmageNet architecture. During the geometry encoding stage (top), each garment is encoded into a set of fixed-size (72-dimensional) latent vectors using a Variational Autoencoder (VAE). These compact latent representations serve as training targets for the subsequent diffusion generation stage (bottom). In the diffusion stage, we employ a diffusion transformer (DiT) denoiser, integrating multi-modal conditions, including line-art sketches, raw sewing patterns, and point clouds via cross-attention mechanisms to effectively guide and control the garment generation process.
• Figure 4: Illustration of stitch representation ambiguity and our point‐wise solution. Existing edge‐based methods suffer from inconsistencies due to arbitrary edge splits: in (a) and (b), the red lines depict the same physical stitch, yet their extracted edge features (shown below) differ in both length and parameter encoding. In contrast, our point‐wise stitching (c) directly anchors stitch correspondences to mesh vertices in physical space, producing consistent, robust sewing relationships independent of panel tessellation.
• Figure 5: Overview of sewing relationship recovery and simulation-ready sewing pattern reconstruction from the generated Garmage (a). Unlike previous edge-based methods, we predict vertex-level sewing relationships. Specifically, we first sample boundary points (c) from the generated Garmage representation. Our GarmageJigsaw takes the boundary points as input, and leverages a point classifier to identify sewing versus non-sewing points (d), followed by a stitch predictor that recovers point-to-point stitches (e), represented as an adjacency matrix. Concurrently, we extract vectorized sewing patterns (b) from the Garmage and transfer the predicted point stitches onto these vectorized patterns (f). We then reconstruct triangle meshes from the vectorized sewing pattern with a Delaunay triangulation constraint by the predicted stitches. Finally, we retrieve vertex-wise draping status from the generated Garmage, leading to a simulation-ready triangle mesh that can be directly integrated into any conventional cloth simulation engine to produce the physically plausible garment (g).