STG-Avatar: Animatable Human Avatars via Spacetime Gaussian

TL;DR

STG-Avatar tackles the rigidity–detail trade-off in monocular avatar reconstruction by integrating LBS-driven global skeletal deformations with Spacetime Gaussians that adaptively model non-rigid, high-frequency details. Optical-flow guided densification concentrates Gaussian points in dynamic regions, while a lightweight MLP decodes color, enabling real-time rendering at 60 FPS and training in about 25 minutes. On ZJU-MoCap and THUman4.0, STG-Avatar achieves state-of-the-art reconstruction quality (e.g., PSNR ≈ 31.6, SSIM ≈ 0.954, LPIPS ≈ 0.038) and robust generalization, outperforming NeRF-based and prior 3DGS baselines. The approach offers practical impact for robotic teleoperation, digital twins, and XR by delivering high-fidelity, animatable avatars from monocular video with efficient training and rendering.

Abstract

Realistic animatable human avatars from monocular videos are crucial for advancing human-robot interaction and enhancing immersive virtual experiences. While recent research on 3DGS-based human avatars has made progress, it still struggles with accurately representing detailed features of non-rigid objects (e.g., clothing deformations) and dynamic regions (e.g., rapidly moving limbs). To address these challenges, we present STG-Avatar, a 3DGS-based framework for high-fidelity animatable human avatar reconstruction. Specifically, our framework introduces a rigid-nonrigid coupled deformation framework that synergistically integrates Spacetime Gaussians (STG) with linear blend skinning (LBS). In this hybrid design, LBS enables real-time skeletal control by driving global pose transformations, while STG complements it through spacetime adaptive optimization of 3D Gaussians. Furthermore, we employ optical flow to identify high-dynamic regions and guide the adaptive densification of 3D Gaussians in these regions. Experimental results demonstrate that our method consistently outperforms state-of-the-art baselines in both reconstruction quality and operational efficiency, achieving superior quantitative metrics while retaining real-time rendering capabilities. Our code is available at https://github.com/jiangguangan/STG-Avatar

Figures (5)

• Figure 1: Using a monocular video, our STG-Avatar can train an animatable 3D Gaussian avatar in just 25 minutes, enabling real-time rendering of novel view synthesis and dynamic pose animation at 60 FPS on an RTX 4090 GPU.
• Figure 2: Framework. Our framework introduces a three-stage hierarchical pipeline for monocular avatar creation: (1) SMPL-guided spatiotemporal Gaussian initialization enabling efficient 3D prior utilization; (2) Rigid-nonrigid co-optimization combining LBS with polynomial deformation fields and flow-guided adaptive sampling; (3) Dynamic-aware neural rendering through lightweight MLP decoding and differentiable rasterization.
• Figure 3: Qualitative Comparison on ZJU-MoCap Dataset peng2021neural. Compared to other state-of-the-art methods, STG-Avatar achieves superior detail in reconstructing the non-rigid parts of the human body.
• Figure 4: Qualitative Comparison on THUman4.0 Dataset zheng2022structured. Our method showcases outstanding rendering capabilities on the THUman4.0 dataset, accurately reconstructing the fine details of human subjects. Every element is rendered with high fidelity, from clothing textures and wrinkles to facial features.
• Figure 5: Ablation Study Visualization on the ZJU-MoCap Dataset peng2021neural. The left half of this figure presents the results obtained without using optical flow-guided sampling, while the right half shows the output of our complete model. It can be observed that the full model demonstrates notable improvements in detailed regions such as hands and faces.