SPARK: Self-supervised Personalized Real-time Monocular Face Capture

TL;DR

This paper proposes a method for high-precision 3D face capture taking advantage of a collection of unconstrained videos of a subject as prior information, and builds on a two stage approach that results in a trained encoder capable of efficiently regressing pose and expression parameters in real-time from previously unseen images.

Abstract

Feedforward monocular face capture methods seek to reconstruct posed faces from a single image of a person. Current state of the art approaches have the ability to regress parametric 3D face models in real-time across a wide range of identities, lighting conditions and poses by leveraging large image datasets of human faces. These methods however suffer from clear limitations in that the underlying parametric face model only provides a coarse estimation of the face shape, thereby limiting their practical applicability in tasks that require precise 3D reconstruction (aging, face swapping, digital make-up, ...). In this paper, we propose a method for high-precision 3D face capture taking advantage of a collection of unconstrained videos of a subject as prior information. Our proposal builds on a two stage approach. We start with the reconstruction of a detailed 3D face avatar of the person, capturing both precise geometry and appearance from a collection of videos. We then use the encoder from a pre-trained monocular face reconstruction method, substituting its decoder with our personalized model, and proceed with transfer learning on the video collection. Using our pre-estimated image formation model, we obtain a more precise self-supervision objective, enabling improved expression and pose alignment. This results in a trained encoder capable of efficiently regressing pose and expression parameters in real-time from previously unseen images, which combined with our personalized geometry model yields more accurate and high fidelity mesh inference. Through extensive qualitative and quantitative evaluation, we showcase the superiority of our final model as compared to state-of-the-art baselines, and demonstrate its generalization ability to unseen pose, expression and lighting.

Figures (8)

• Figure 1: Illustration of our-two stage adaptation process. In stage 1, we rely on a collection of different video sources of the same person to build a personalized geometry decoder through inverse rendering. In stage 2, the 3DMM of a generalizable feedforward face capture network is swapped with the new decoder, and the encoder is tuned by reconstructing the same adaptation video frames leveraging the pre-estimated reflectance function for each video.
• Figure 2: Comparison to multiple state of the art face reconstruction methods. From left to right: input image, DECA feng2021learning, EMOCA danecek2022EMOCA, EMOCA fine-tuned, SMIRK retsinasSMIRK3DFacial2024 and SPARK. See Fig. \ref{['fig:qualitative_more']} for more examples. More results are also provided in our supplemental videos.
• Figure 3: Illustration of our semantic Intersection-over-Union metric. Left: manually annotated semantic masks for FLAME. Right: two examples with ground-truth segmentation, a render of EMOCA's tracked mesh and our tracked mesh.
• Figure 4: Illustration of our image warping metric. The background and hair are masked out in both frames. We use the tracked geometry to backtrack each rasterized pixel of image $I_t$ to a pixel in image $I_{t-k}$. The white area is the occlusion mask, where we do not compare pixels.
• Figure 5: Examples of reconstructed canonical geometry from MultiFLARE for an increasing number of training sequences.