CloDS: Visual-Only Unsupervised Cloth Dynamics Learning in Unknown Conditions

TL;DR

This work tackles cloth dynamics grounding (CDG), the problem of unsupervised cloth dynamics learning from multi-view visual observations under unknown conditions, by modeling $p(M_{t+1}|M_t)$ and $p(Y_{t+1}|Y_{1:t})$. It introduces CloDS, a three-stage framework that grounds 2D observations into a differentiable 3D cloth mesh via Spatial Mapping Gaussian Splatting with dual-position opacity modulation, and then trains a Graph Neural Dynamics Learner to approximate $p(M_{t+1}|M_t)$ for forward simulation and rendering of $p(Y_{t+1}|Y_{1:t})$. The key contributions are the Cloth Dynamics Grounding formalism, the SMGS operator with world-space and mesh-space conditioning, and an unsupervised training regime that achieves strong generalization to unseen shapes/textures and real garments, outperforming vision-based baselines in dynamic novel-view synthesis and DVC forward prediction. By eliminating the need for physics-based supervision, this approach enables realistic garment animation and robotics in unknown environments using purely visual data, with potential to extend to complex multi-object scenes.

Abstract

Deep learning has demonstrated remarkable capabilities in simulating complex dynamic systems. However, existing methods require known physical properties as supervision or inputs, limiting their applicability under unknown conditions. To explore this challenge, we introduce Cloth Dynamics Grounding (CDG), a novel scenario for unsupervised learning of cloth dynamics from multi-view visual observations. We further propose Cloth Dynamics Splatting (CloDS), an unsupervised dynamic learning framework designed for CDG. CloDS adopts a three-stage pipeline that first performs video-to-geometry grounding and then trains a dynamics model on the grounded meshes. To cope with large non-linear deformations and severe self-occlusions during grounding, we introduce a dual-position opacity modulation that supports bidirectional mapping between 2D observations and 3D geometry via mesh-based Gaussian splatting in video-to-geometry grounding stage. It jointly considers the absolute and relative position of Gaussian components. Comprehensive experimental evaluations demonstrate that CloDS effectively learns cloth dynamics from visual data while maintaining strong generalization capabilities for unseen configurations. Our code is available at https://github.com/whynot-zyl/CloDS. Visualization results are available at https://github.com/whynot-zyl/CloDS_video}.%\footnote{As in this example.

Figures (15)

• Figure 1: CloDS has the following capabilities:(a). Learning underlying cloth dynamics. (b). Video prediction through the forward process of DVC. (c). Novel view synthesis in dynamic scenes.
• Figure 2: Overview of Differentiable Visual Computing framework in Cloth Dynamics Reasoning.
• Figure 3: The overview of Cloth Dynamics Splatting (CloDS) for Cloth Dynamics Grounding. (a). Overall model architecture. (b). The forward and backward processes of Spatial Mapping Gaussian Splatting (SMGS) and DVC forward process. (c). The detail of SMGS forward process. (d). The Mesh Gaussian Splatting in SMGS. SMGS can obtain the mesh of cloth $\tilde{M}_{t+1}$ based on $M_{t}$ or $\tilde{M}_{t}$ through backpropagation. In the forward stage of DVC, $\Delta x^W_t$ is typically set to 0. However, when extracting the mesh $\tilde{M}_{t+1}$ from images $I_{t+1}$, the learnable $\Delta x^W_t$ is used to make $\tilde{I}_{t+1}$ close to $I_{t+1}$.
• Figure 4: The result of rendering. Videos are available in Part 2 at https://github.com/whynot-zyl/CloDS_video.
• Figure 5: Visualization of the cloth prediction results. $t=2$ denotes the first predicted frame, $t=300$ is the last frame of interpolation, and $t=400$ is the last frame of extrapolation. $x^W_t$ and $x^M_t$ represent the 3D and 2D mesh positions, respectively. Errors are visualized in red, with deeper color indicating larger errors. All methods share the same error bars. "Viewed" and "Unviewed" denote whether the cloth’s initial state was seen during training.