ReWeaver: Towards Simulation-Ready and Topology-Accurate Garment Reconstruction

TL;DR

ReWeaver addresses the need for topology-accurate garment reconstruction from sparse multi-view imagery by jointly recovering 3D garment topology and 2D sewing patterns, yielding simulation-ready assets suitable for physical simulation and manipulation. It introduces a VGGT-inspired visual encoder, a bi-path Transformer for 3D curve/patch prediction and connectivity, and a 2D pattern module that preserves explicit 2D–3D correspondences through a set of learned hyper-networks. The approach is trained on GCD-TS, a large-scale textured dataset with explicit pattern annotations, and demonstrates superior topology accuracy, geometry alignment, and seam–panel consistency over baselines. This work advances the simulatable garment reconstruction pipeline, enabling more faithful digital humans, virtual try-on, and robot-assisted garment manipulation.

Abstract

High-quality 3D garment reconstruction plays a crucial role in mitigating the sim-to-real gap in applications such as digital avatars, virtual try-on and robotic manipulation. However, existing garment reconstruction methods typically rely on unstructured representations, such as 3D Gaussian Splats, struggling to provide accurate reconstructions of garment topology and sewing structures. As a result, the reconstructed outputs are often unsuitable for high-fidelity physical simulation. We propose ReWeaver, a novel framework for topology-accurate 3D garment and sewing pattern reconstruction from sparse multi-view RGB images. Given as few as four input views, ReWeaver predicts seams and panels as well as their connectivities in both the 2D UV space and the 3D space. The predicted seams and panels align precisely with the multi-view images, yielding structured 2D--3D garment representations suitable for 3D perception, high-fidelity physical simulation, and robotic manipulation. To enable effective training, we construct a large-scale dataset GCD-TS, comprising multi-view RGB images, 3D garment geometries, textured human body meshes and annotated sewing patterns. The dataset contains over 100,000 synthetic samples covering a wide range of complex geometries and topologies. Extensive experiments show that ReWeaver consistently outperforms existing methods in terms of topology accuracy, geometry alignment and seam-panel consistency.

Figures (8)

• Figure 1: ReWeaver. From as few as four input views, ReWeaver reconstructs high-precision sewing patterns with complex topology together with their corresponding 3D geometry. The method outputs a unified 2D--3D garment representation, where each panel and edge is explicitly linked to its associated 3D points. This enables faithful, simulation-ready garment assets to be recovered from ordinary and sparse-view photographs without a controlled capture setup. The reconstructed garment geometry and topology is precisely aligned with the input images and can be used for 3D structural perception.
• Figure 2: Visualization of the terminologies used in this paper.
• Figure 3: Pipeline of our method. Our VGGT-like image encoder extracts features from multi-view images (Section \ref{['subsec:visual:encoder']}), which then interact with predefined patch and curve queries. In the 3D geometry and topology prediction module (Section \ref{['subsec:curve:patch:prediction']}), these queries and the image tokens pass through stacked self- and cross-attention blocks. The resulting tokens are decoded into 3D curves and patches. The same tokens are then reused in the 2D pattern prediction module (Section \ref{['subsec:flattening']}): guided by the refined topology, we group the patches and curves into patch-centric groups, apply intra-group attention, and finally decode the edges of the 2D panels. The decoded 2D panels can be directly used for physical simulation.
• Figure 4: Texture differences between GCD and GCD-TS. Using the same garment geometry, GCD textures reveal strong seam cues (e.g., highlighted regions), which are unrealistic and can lead to overfitting. GCD-TS replaces these with more realistic, diverse textures to improve generalization.
• Figure 5: Visualization on adaptive sampling density at inference time. From left to right, the point set sampled at a fixed density of $20\times20$ points per patch; the point set sampled at the adaptive-density; the ground truth point cloud.