Real-Time Animatable 2DGS-Avatars with Detail Enhancement from Monocular Videos

TL;DR

This work tackles real-time animatable avatar reconstruction from monocular video by leveraging 2D Gaussian Splatting (2DGS) embedded into the SMPL mesh. A Rotation Compensation Network (RCN) learns rotation residuals that capture non-rigid deformations, enabling stable, high-fidelity pose-driven animation. Key contributions include dense joint-region primitve embedding with offset surface residuals, forward LBS-based posed-space transformation, and a multi-term loss with scale/joint regularization to constrain deformations. Experiments on public datasets show superior novel-view and novel-pose synthesis and faster mesh reconstruction compared with prior methods, highlighting practical potential for real-time animation in games, AR, and social media.

Abstract

High-quality, animatable 3D human avatar reconstruction from monocular videos offers significant potential for reducing reliance on complex hardware, making it highly practical for applications in game development, augmented reality, and social media. However, existing methods still face substantial challenges in capturing fine geometric details and maintaining animation stability, particularly under dynamic or complex poses. To address these issues, we propose a novel real-time framework for animatable human avatar reconstruction based on 2D Gaussian Splatting (2DGS). By leveraging 2DGS and global SMPL pose parameters, our framework not only aligns positional and rotational discrepancies but also enables robust and natural pose-driven animation of the reconstructed avatars. Furthermore, we introduce a Rotation Compensation Network (RCN) that learns rotation residuals by integrating local geometric features with global pose parameters. This network significantly improves the handling of non-rigid deformations and ensures smooth, artifact-free pose transitions during animation. Experimental results demonstrate that our method successfully reconstructs realistic and highly animatable human avatars from monocular videos, effectively preserving fine-grained details while ensuring stable and natural pose variation. Our approach surpasses current state-of-the-art methods in both reconstruction quality and animation robustness on public benchmarks.

Figures (11)

• Figure 1: We perform SMPL joint-prior guided non-uniform sampling across the mesh surface. Each 2D Gaussian primitive is embedded into a sampled triangle using barycentric coordinates $(u,v)$ and an offset distance $d$ measured along the triangle’s normal $\mathbf{n}$. We then deform the mesh via Linear Blend Skinning to transfer the 2D Gaussians primitives from the canonical space to the posed space and apply our proposed RCN to refine their orientations. Finally, we jointly optimize the Gaussian embedding parameters and the RCN weights through differentiable rasterization.
• Figure 2: The dynamic update mechanism of the barycentric coordinates for adjacent triangle transfer.
• Figure 3: Rigid mesh deformation creates rotation errors (green). RCN learns the rotation residual (red) and corrects it (blue), realigning Gaussians to the true surface.
• Figure 4: Comparisons on the PeopleSnapshot alldieck2018detailed and Synthetic jiang2022selfrecon datasets. Rows 1–2 show reconstruction results on the PeopleSnapshot dataset, and Rows 3–4 present comparisons on the Synthetic dataset. From left to right, the columns correspond to Ground Truth, our method, SplattingAvatar shao2024splattingavatar, GaussianAvatar hu2024gaussianavatar, and Gart lei2024gart.
• Figure 5: Qualitative comparison of our method with three state-of-the-art methods under novel pose conditions. From left to right: our method, SplattingAvatar shao2024splattingavatar, GaussianAvatar hu2024gaussianavatar, and Gart lei2024gart.