Learning 3D Garment Animation from Trajectories of A Piece of Cloth

TL;DR

This work tackles data efficiency and generalization in garment animation by disentangling constitutive-law learning from dynamics. It introduces Energy Unit Network (EUNet) to directly learn energy-based constitutive laws from trajectories of a single piece of cloth, decomposing energy into edge-wise units that capture stretching and bending without relying on analytical priors or differentiable simulators. EUNet is regularized with a vertex-wise contrastive loss to produce robust energy gradients, and its learned energy is integrated into an energy-optimization framework to animate diverse garments via backward-Euler dynamics. Experiments show that EUNet-constrained simulators outperform garment-wise supervised baselines and physics-prior methods in terms of accuracy, stability, and plausibility, demonstrating data-efficient, topology-agnostic garment animation with strong generalization to unseen garments.

Abstract

Garment animation is ubiquitous in various applications, such as virtual reality, gaming, and film producing. Recently, learning-based approaches obtain compelling performance in animating diverse garments under versatile scenarios. Nevertheless, to mimic the deformations of the observed garments, data-driven methods require large scale of garment data, which are both resource-wise expensive and time-consuming. In addition, forcing models to match the dynamics of observed garment animation may hinder the potentials to generalize to unseen cases. In this paper, instead of using garment-wise supervised-learning we adopt a disentangled scheme to learn how to animate observed garments: 1). learning constitutive behaviors from the observed cloth; 2). dynamically animate various garments constrained by the learned constitutive laws. Specifically, we propose Energy Unit network (EUNet) to model the constitutive relations in the format of energy. Without the priors from analytical physics models and differentiable simulation engines, EUNet is able to directly capture the constitutive behaviors from the observed piece of cloth and uniformly describes the change of energy caused by deformations, such as stretching and bending. We further apply the pre-trained EUNet to animate various garments based on energy optimizations. The disentangled scheme alleviates the need of garment data and enables us to utilize the dynamics of a piece of cloth for animating garments. Experiments show that while EUNet effectively delivers the energy gradients due to the deformations, models constrained by EUNet achieve more stable and physically plausible performance comparing with those trained in garment-wise supervised manner. Code is available at https://github.com/ftbabi/EUNet_NeurIPS2024.git .

Figures (9)

• Figure 1: Given the observed piece of cloth as shown on the left, we aim to animate various garments inheriting the attributes from the observations as shown in the middle and right. We disentangle the garment-wise learning into two sub-tasks: 1) learning constitutive relations by our proposed EUNet; 2) animating diverse garments through energy optimization constrained by EUNet.
• Figure 2: Overview of the disentangled learning scheme and our EUNet for garment animation. Unlike traditional garment-wise learning which relies on large scale of garment data, we first aim to capture the constitutive relations from the observed piece of cloth using our EUNet. Without the prior of analytical clothing models or differentiable simulator, EUNet is able to extract the potential energies of the cloth under different deformations, such as stretching and bending, directly from the observed trajectories in a system with dissipation. Secondly, given the external force sequences and the garment templates, we dynamically animate various garments based on the energy optimizations, where EUNet serves as material priors. As a result, we can animate garments that inherit the attributes, such as the stiffness, from the observed cloth, and achieve robust and physically plausible animations.
• Figure 3: Visualization of the potential energies predicted by our EUNet either without the dissipation energy branch $\Phi_d$ or the contrastive loss term $\mathcal{L}_{\mathrm{con}}$. We sample the materials of silk and leather for demonstration, and change both the edge length and angles between vertex normals to verify the energy gradients caused by stretching and bending. Since silk is easier to bend, the energy gradients caused by different angles are smaller than those of leather. Both the dissipation energy branch and the contrastive loss enable EUNet to obtain reasonable energy gradients due to the deformations.
• Figure 4: Qualitative results by our disentangled training scheme. We train MGN-S and MGN-H constrained by our EUNet through energy optimization scheme. Since the observed cloth to train EUNet is made of the same materials as the ground truth garments, the constitutive relations captured by EUNet are consistent with the ground truth data. As a result, MGN-S and MGN-H constrained by EUNet deliver similar deformation patterns as the ground truth garments without accessing any garment data. Even in long-term predictions, we can obtain plausible wrinkles, which are difficult for models trained in a garment-wise learning pipeline, and robust interactions with the human body.
• Figure 5: Qualitative results of garment animations by baselines. While the garment-wise learning scheme enables the baselines to obtain reasonable predictions within a short period of time, the errors increase for long-term predictions. Though we estimate the physics parameters required by the analytical clothing models, MGN+PHYS and HOOD generate overly soft garments and struggle to mimic the deformations of the ground truth data.