Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GVA: Reconstructing Vivid 3D Gaussian Avatars from Monocular Videos

Xinqi Liu, Chenming Wu, Jialun Liu, Xing Liu, Jinbo Wu, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang

TL;DR

A pose refinement technique is introduced to improve hand and foot pose accuracy by aligning normal maps and silhouettes and addressing the problems of unbalanced aggregation and initialization bias that previously diminished the quality of 3D Gaussian avatars through a novel surface-guided re-initialization method.

Abstract

In this paper, we present a novel method that facilitates the creation of vivid 3D Gaussian avatars from monocular video inputs (GVA). Our innovation lies in addressing the intricate challenges of delivering high-fidelity human body reconstructions and aligning 3D Gaussians with human skin surfaces accurately. The key contributions of this paper are twofold. Firstly, we introduce a pose refinement technique to improve hand and foot pose accuracy by aligning normal maps and silhouettes. Precise pose is crucial for correct shape and appearance reconstruction. Secondly, we address the problems of unbalanced aggregation and initialization bias that previously diminished the quality of 3D Gaussian avatars, through a novel surface-guided re-initialization method that ensures accurate alignment of 3D Gaussian points with avatar surfaces. Experimental results demonstrate that our proposed method achieves high-fidelity and vivid 3D Gaussian avatar reconstruction. Extensive experimental analyses validate the performance qualitatively and quantitatively, demonstrating that it achieves state-of-the-art performance in photo-realistic novel view synthesis while offering fine-grained control over the human body and hand pose. Project page: https://3d-aigc.github.io/GVA/.

GVA: Reconstructing Vivid 3D Gaussian Avatars from Monocular Videos

TL;DR

A pose refinement technique is introduced to improve hand and foot pose accuracy by aligning normal maps and silhouettes and addressing the problems of unbalanced aggregation and initialization bias that previously diminished the quality of 3D Gaussian avatars through a novel surface-guided re-initialization method.

Abstract

In this paper, we present a novel method that facilitates the creation of vivid 3D Gaussian avatars from monocular video inputs (GVA). Our innovation lies in addressing the intricate challenges of delivering high-fidelity human body reconstructions and aligning 3D Gaussians with human skin surfaces accurately. The key contributions of this paper are twofold. Firstly, we introduce a pose refinement technique to improve hand and foot pose accuracy by aligning normal maps and silhouettes. Precise pose is crucial for correct shape and appearance reconstruction. Secondly, we address the problems of unbalanced aggregation and initialization bias that previously diminished the quality of 3D Gaussian avatars, through a novel surface-guided re-initialization method that ensures accurate alignment of 3D Gaussian points with avatar surfaces. Experimental results demonstrate that our proposed method achieves high-fidelity and vivid 3D Gaussian avatar reconstruction. Extensive experimental analyses validate the performance qualitatively and quantitatively, demonstrating that it achieves state-of-the-art performance in photo-realistic novel view synthesis while offering fine-grained control over the human body and hand pose. Project page: https://3d-aigc.github.io/GVA/.
Paper Structure (18 sections, 9 equations, 12 figures, 4 tables)

This paper contains 18 sections, 9 equations, 12 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Human Avatar Reconstruction
  4. Human Avatar Representation
  5. Preliminary
  6. Proposed Method
  7. The Representation of 3D Gaussian Avatars
  8. Pose Refinement for Avatar Reconstruction
  9. Surface-Guided Gaussian Re-Initialization
  10. Differentiable Rendering Loss Function
  11. Experiments
  12. Setup and Datasets
  13. Baselines and Evaluation Metrics
  14. Qualitative Experiments
  15. Quantitative Results
  16. ...and 3 more sections

Figures (12)

  • Figure 1: Our proposed GVA enables the effective reconstruction of 3D Gaussian avatars from monocular videos. Its capability for flexible pose adjustments via external motions results in realistic avatars.
  • Figure 2: The illustration of the widespread phenomena of unbalanced aggregation and initialization bias within the 3D Gaussian avatar reconstruction algorithms.
  • Figure 3: The framework utilizes a monocular video to obtain refined body and hand poses. The Gaussian avatar model is adjusted based on the whole-body skeleton to match the pose in the image. Consistency with image observations is maintained through differentiable rendering and optimization of Gaussian properties. An surface-guided re-initialization mechanism enhances rendering quality and Gaussian point distribution. The model can adapt to new poses from videos or generated sequences.
  • Figure 4: The surface-guided re-initialization mechanism uses the three operations of Meshing, Resampling, and Re-Gaussian to redistribute unevenly Gaussian points near the real surface, thereby enhancing the stability of the avatar in novel poses.
  • Figure 5: Rendered frames of our reconstructed Gaussian avatar from novel views.
  • ...and 7 more figures