SplatFace: Gaussian Splat Face Reconstruction Leveraging an Optimizable Surface

TL;DR

The proposed SplatFace method is designed to simultaneously deliver both high-quality novel view rendering and accurate 3D mesh reconstructions and is competitive with both other Gaussian splatting techniques in novel view synthesis and other 3D reconstruction methods in producing 3D face meshes with high geometric precision.

Abstract

We present SplatFace, a novel Gaussian splatting framework designed for 3D human face reconstruction without reliance on accurate pre-determined geometry. Our method is designed to simultaneously deliver both high-quality novel view rendering and accurate 3D mesh reconstructions. We incorporate a generic 3D Morphable Model (3DMM) to provide a surface geometric structure, making it possible to reconstruct faces with a limited set of input images. We introduce a joint optimization strategy that refines both the Gaussians and the morphable surface through a synergistic non-rigid alignment process. A novel distance metric, splat-to-surface, is proposed to improve alignment by considering both the Gaussian position and covariance. The surface information is also utilized to incorporate a world-space densification process, resulting in superior reconstruction quality. Our experimental analysis demonstrates that the proposed method is competitive with both other Gaussian splatting techniques in novel view synthesis and other 3D reconstruction methods in producing 3D face meshes with high geometric precision.

Figures (7)

• Figure 1: Diagram of overall process: SplatFace begins by initializing 3D Gaussians and a surface using the template mesh derived from a 3D Morphable Model (3DMM). The Gaussians and surface are then simultaneously refined through a joint optimization process. Splats are constrained by image photoconsistency, and by a novel splat-to-surface distance metric. This measure is introduced to accurately quantify the discrepancies between the Gaussian splat distribution and the surface, considering both the position and covariance of the Gaussians. Additionally, the presence of a surface allows the introduction of world-space densification. As a result of this overall process, we obtain both enhanced novel view synthesis and a finely-tuned 3D mesh.
• Figure 2: Joint optimization of Gaussians and geometric surface via splat-to-surface distance. (a) Illustration of splat-to-surface distance and (b) the modifications to splats and surface this distance is meant to encourage. Point-to-surface distance, shown in red, ignores covariance and considers each splat as a point. The point-to-surface distance will only calculate distance from splat center to the surface. In contrast, splat-to-surface distance accounts for the extended nature of the Gaussian distribution. Minimizing splat-to-surface distance will simultaneously optimize the orientation, position and scale of Gaussian Splats.
• Figure 3: World-space densification. We densify splats that are far from the surface using the proposed splat-to-surface distance. Gaussians with any sampled points far from the surface are considered far and either split or cloned. This helps generate multiple splats closer to the surface which are more expressive than a single floating or spiky splat which extends far from the surface.
• Figure 4: Qualitative comparison of face shape estimation with state of the art methods using the FaceScape dataset. The error maps visualize a range from blue (0 mm) to yellow (3 mm). Our method produces a smooth reconstruction with lower error.
• Figure 5: Qualitative comparison on novel view synthesis. We compare our results with other Gaussian splatting methods on the FaceScapeyang2020facescape, ILSHzheng2023ilsh and NeRSemblekirschstein2023nersemble datasets. Each method uses few-view input images, shown on the left. Our method produce results with fewer artifacts than the comparison methods.