Gaussian Garments: Reconstructing Simulation-Ready Clothing with Photorealistic Appearance from Multi-View Video

TL;DR

Gaussian Garments is introduced, a novel approach for reconstructing realistic simulation-ready garment assets from multi-view videos that represents garments with a combination of a 3D mesh and a Gaussian texture that encodes both the color and high-frequency surface details.

Abstract

We introduce Gaussian Garments, a novel approach for reconstructing realistic simulation-ready garment assets from multi-view videos. Our method represents garments with a combination of a 3D mesh and a Gaussian texture that encodes both the color and high-frequency surface details. This representation enables accurate registration of garment geometries to multi-view videos and helps disentangle albedo textures from lighting effects. Furthermore, we demonstrate how a pre-trained graph neural network (GNN) can be fine-tuned to replicate the real behavior of each garment. The reconstructed Gaussian Garments can be automatically combined into multi-garment outfits and animated with the fine-tuned GNN.

Figures (14)

• Figure 1: We introduce Gaussian Garments, a novel approach for reconstructing realistic simulation-ready garments from multi-view videos. Our natural coupling of 3D meshes and 3D Gaussian splatting allows Gaussian Garments to accurately represent both the overall geometry and the high-frequency details of human clothing. The reconstructed garments can then be retargeted to novel human models, resized to fit novel body shapes, and simulated over moving bodies with novel motions. Our approach also enables the automatic construction of complex multi-layer outfits from a set of separately captured Gaussian garments.
• Figure 2: The procedure for obtaining simulation-ready photorealistic garment assets consists of four steps. In Step 1, we initialize the garment's geometry and appearance from a single multi-view frame (Sec. \ref{['sec:initialize']}). In Step 2, we register the garment geometry to multi-view videos (Sec. \ref{['sec:registration']}). In Step 3, we optimize the garment's appearance over the training sequences. In Step 4, we fine-tune a simulation GNN to accurately replicate the garment's real behavior. The resulting garment assets can be directly simulated with the GNN, combined into multi-garment outfits, and resized to fit different body shapes.S
• Figure 3: To register the garment mesh we render the Gaussians rigidly attached to the mesh faces (top left) and optimize a combination of the RGB loss $\mathcal{L}_{\textit{RGB}}$ and physical energies $\mathcal{L}_{\textit{phys}}$. We also use a body penetration term $\mathcal{L}_{\textit{body}}$ to ensure that the garment conforms to the body model.
• Figure 4: We model the appearance of Gaussian Garments using a combination of an albedo Gaussian texture and a neural network that predicts lighting effects and local translational offsets. The albedo Gaussian texture stores color information along with Gaussian parameters, including local rotation, translation, and scale. During rendering we regularly sample the Gaussian texture and spawn the 3D Gaussians rigidly attached to the garment surface.
• Figure 5: We disentangle the albedo color of the Gaussian Garments from the lighting effects predicted by a neural network. Here we show four garments rendered over the registered sequence. Note how, when rendered with albedo colors, the garments lack any shadows or specular effects. The lighting information comes solely from network predictions and matches the ground-truth information. The figure shows registered mesh sequences that were not seen by the appearance model during training.