PGC: Physics-Based Gaussian Cloth from a Single Pose

TL;DR

This work tackles the challenge of reconstructing photorealistic, simulation-ready garments from a single static pose captured across multiple views. It introduces a hybrid representation that combines mesh-embedded 3D Gaussian splats for near-field detail with a cloth-specific PBR shading model for far-field illumination, fused through a Gaussian-PBR hybrid renderer to produce novel-pose renders. The approach enables realistic garment simulation and relighting without requiring multi-frame tracking, achieving improved detail and shading fidelity over state-of-the-art baselines. By enabling physically plausible deformations and real-time-friendly rendering, this method advances realistic avatar garments for virtual try-on and telepresence applications.

Abstract

We introduce a novel approach to reconstruct simulation-ready garments with intricate appearance. Despite recent advancements, existing methods often struggle to balance the need for accurate garment reconstruction with the ability to generalize to new poses and body shapes or require large amounts of data to achieve this. In contrast, our method only requires a multi-view capture of a single static frame. We represent garments as hybrid mesh-embedded 3D Gaussian splats, where the Gaussians capture near-field shading and high-frequency details, while the mesh encodes far-field albedo and optimized reflectance parameters. We achieve novel pose generalization by exploiting the mesh from our hybrid approach, enabling physics-based simulation and surface rendering techniques, while also capturing fine details with Gaussians that accurately reconstruct garment details. Our optimized garments can be used for simulating garments on novel poses, and garment relighting. Project page: https://phys-gaussian-cloth.github.io .

Figures (12)

• Figure 1: We present a method to recover photorealistic, simulation-ready garments from a multi-view capture of a single static pose. The recovered garments consist of simulatable geometry and fine-detail appearance. Moreover, our results generalize to novel motion as they can be simulated on human motion sequences, and our garments are relightable.
• Figure 2: Method Overview. Given a multi-view capture of a clothed human in a single pose, we first extract the garment mesh and fit a 3DGS representation with mesh-embedded Gaussian splats. Simultaneously, we fit an albedo map of the ground truth image and back-project onto the mesh to generate a textured mesh. At inference, the mesh is shaded with a physically-based shading model and the resulting mesh colors are then transferred to zero-order spherical harmonics on the pre-optimized splats. Finally, we combine the high-pass of the original Gaussian splat reconstruction with the low-pass of the traditionally shaded result to produce the final render.
• Figure 3: PBR model evaluation. Each mesh is shaded with 3 different shading models and compared with ground truth (a). Notice the sheen around the cardigan sleeve and the wrinkle near the fleece abdomen denoted by the red square. The Lambertian model (b) shows no sheen, Disney BRDF (c) over-estimates the forward scattered sheen, whereas our model (d) produces the best match to ground truth sheen. PSNR values are computed to quantitatively demonstrate the improvement of our model over prior methods.
• Figure 4: Reconstruction. The fleece garment $\bm{I}$ is reconstructed from the sum of the far-field approximation $l(\bm{S})$ and near-field approximation $h(\bm{G})$ to produce the final reconstruction $\hat{\bm{I}}$. The image on the right shows the reconstruction error.
• Figure 5: Ablation. We compare our approach to different ablations of our method and ground truth (a) of a novel pose. 3DGS-Only (b) has baked in shading from the training pose, resulting in unrealistic appearance when deformed to novel poses. PBR-Only (c) has reshading, but lacks detailed shading, and can only model flat 2D texture colors. Ours (d) has the best of both worlds.