RMAvatar: Photorealistic Human Avatar Reconstruction from Monocular Video Based on Rectified Mesh-embedded Gaussians

TL;DR

RMAvatar presents a photorealistic human avatar reconstruction method from monocular video using Gaussian splatting embedded on a mesh. It combines explicit mesh geometry for motion and shape with implicit appearance rendering via Gaussian splatting. The approach includes two modules, Gaussian initialization and Gaussian rectification, embedding Gaussians into triangular faces and moving them with the mesh to capture low-frequency motion and surface deformation. To address limitations of linear skinning for non-rigid transformations, a pose-related Gaussian rectification module learns fine-grained deformations, enhancing realism and expressiveness. Experiments on public datasets show state-of-the-art rendering quality and quantitative performance, with a project page provided for reference.

Abstract

We introduce RMAvatar, a novel human avatar representation with Gaussian splatting embedded on mesh to learn clothed avatar from a monocular video. We utilize the explicit mesh geometry to represent motion and shape of a virtual human and implicit appearance rendering with Gaussian Splatting. Our method consists of two main modules: Gaussian initialization module and Gaussian rectification module. We embed Gaussians into triangular faces and control their motion through the mesh, which ensures low-frequency motion and surface deformation of the avatar. Due to the limitations of LBS formula, the human skeleton is hard to control complex non-rigid transformations. We then design a pose-related Gaussian rectification module to learn fine-detailed non-rigid deformations, further improving the realism and expressiveness of the avatar. We conduct extensive experiments on public datasets, RMAvatar shows state-of-the-art performance on both rendering quality and quantitative evaluations. Please see our project page at https://rm-avatar.github.io.