PBDyG: Position Based Dynamic Gaussians for Motion-Aware Clothed Human Avatars

TL;DR

PBDyG tackles the challenge of reconstructing animatable clothed humans from multiview videos by modeling clothing as 3D Gaussians tied to a SMPL body and simulating movement-dependent cloth dynamics with position based dynamics. The approach combines Dynamic Gaussian Splatting for tracking, SMPL refinement for accurate body geometry, and XPBD-enabled PBD to capture non-rigid garment deformations while preserving body-cloth contact. Key contributions include a three-stage pipeline (Gaussian reconstruction/tracking, SMPL refinement, and PBD parameter learning), a sampling/interpolation strategy to manage large Gaussian sets, and stabilization techniques such as AirMesh constraints. The results demonstrate realistic, reanimatable avatars with highly deformable garments, along with new evaluation metrics (HF-SSIM, HF-PSNR) that better reflect perceptual fidelity for clothing details.

Abstract

This paper introduces a novel clothed human model that can be learned from multiview RGB videos, with a particular emphasis on recovering physically accurate body and cloth movements. Our method, Position Based Dynamic Gaussians (PBDyG), realizes ``movement-dependent'' cloth deformation via physical simulation, rather than merely relying on ``pose-dependent'' rigid transformations. We model the clothed human holistically but with two distinct physical entities in contact: clothing modeled as 3D Gaussians, which are attached to a skinned SMPL body that follows the movement of the person in the input videos. The articulation of the SMPL body also drives physically-based simulation of the clothes' Gaussians to transform the avatar to novel poses. In order to run position based dynamics simulation, physical properties including mass and material stiffness are estimated from the RGB videos through Dynamic 3D Gaussian Splatting. Experiments demonstrate that our method not only accurately reproduces appearance but also enables the reconstruction of avatars wearing highly deformable garments, such as skirts or coats, which have been challenging to reconstruct using existing methods.

Figures (8)

• Figure 1: Position Based Dynamic Gaussians (PBDyG) reconstructs a person from multiview videos such that their clothing deforms in accordance with the movement, not just rigid pose, of the person. This makes possible reconstruction of animatable avatars of people wearing highly deformable loose clothes. Example test animations are shown.
• Figure 2: Overview of our method. PBDyG learning consists of three main steps: 3D Gaussian reconstruction and tracking, SMPL fitting and refinement also referencing the Gaussians, and PBD parameter estimation.
• Figure 3: Reconstruction comparison for Subject 0166. Our method achieves more accurate visual results yet score poorly on conventional metrics which do not reflect perceptual quality well. We believe our new metric is more faithful to the qualitative results.
• Figure 4: Reconstruction comparison for Subject 0206. Our results have the same tendency (high qualitative accuracy, low quantitive results in SSIM and LPIPS).
• Figure 5: Example avatars reconstructed by our method. For each subject, the result on the left is in the training pose. The remaining columns are the avatars reanimated to two novel poses per example.