DiffAvatar: Simulation-Ready Garment Optimization with Differentiable Simulation

TL;DR

DiffAvatar addresses the challenge of generating simulation-ready avatar assets from a single scan by jointly optimizing body shape/pose, garment 2D patterns, and material properties within a differentiable cloth simulation framework. It introduces a differentiable control cage to regularize 2D garment patterns and relies on a minimal garment template library to reconstruct both geometry and physics parameters from noisy scans. The approach yields high-quality, physically plausible drapes and compatible 3D/2D representations suitable for physics-based applications, demonstrated by quantitative improvements over baselines and the ability to generate novel simulated sequences. This work enables scalable, personalized avatar asset creation for telepresence and other applications requiring realistic clothing physics.

Abstract

The realism of digital avatars is crucial in enabling telepresence applications with self-expression and customization. While physical simulations can produce realistic motions for clothed humans, they require high-quality garment assets with associated physical parameters for cloth simulations. However, manually creating these assets and calibrating their parameters is labor-intensive and requires specialized expertise. Current methods focus on reconstructing geometry, but don't generate complete assets for physics-based applications. To address this gap, we propose \papername,~a novel approach that performs body and garment co-optimization using differentiable simulation. By integrating physical simulation into the optimization loop and accounting for the complex nonlinear behavior of cloth and its intricate interaction with the body, our framework recovers body and garment geometry and extracts important material parameters in a physically plausible way. Our experiments demonstrate that our approach generates realistic clothing and body shape suitable for downstream applications. We provide additional insights and results on our webpage: https://people.csail.mit.edu/liyifei/publication/diffavatar/

Figures (9)

• Figure 1: We present DiffAvatar, an automated computational method to recover simulation-ready garment and body assets. Starting from a multi-view capture, we reconstruct a semantically segmented 3D mesh. The segmented clothing geometry acts as a target shape for our optimization pipeline. Our method recovers body shape and pose, clothing pattern and clothing material parameters from a single scan. We optimize a clothing template in 2D pattern space to reproduce the captured clothing in 3D in a physical way. We compute gradients of required parameters using a differentiable simulation approach.
• Figure 2: DiffAvatar generates simulation-ready avatar assets from inputs obtained through a multi-view capture. Our pipeline initially preprocesses the 3D scan to segment the target garment and establish the initial pose and shape of the parametric body model. We employ a differentiable simulation framework to align our simulated garment with the segmented garment by jointly optimizing the garment's design and material parameters, along with the body shape.
• Figure 3: 3D garments (right) can be compacted represented as their 2D panels (left). Seams are visualized as dotted-lines.
• Figure 4: 2D pattern comparison. The automatically optimized 2D patterns of the dress (first row) and long sleeve shirt (second row) by DiffAvatar closely match the manually created artist ones. However, those generated by NeuralTailor Korosteleva2022 do not resemble the artist-made patterns closely and miss important details.
• Figure 5: Body shape and cloth material estimation. Left: We fit a statistical body model to the 3D scan and refine this estimate using our differentiable simulation pipeline and show the difference in shape between initial and refined in black. Right: Our initial material estimate produces large folds that do not match the scan as well as our optimized result shown rightmost.