Pix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image

TL;DR

Pix2NPHM presents a first-of-its-kind feed-forward regressor that outputs Neural Parametric Head Model (NPHM) latent codes from a single image, enabling high-fidelity 3D face reconstruction with fast inference. It leverages geometry-aware Vision Transformer backbones pretrained on per-pixel geometric tasks, trains on a large mix of 3D-registered data and 2D video data with self-supervised normal losses, and augments with test-time optimization for further fidelity. The approach achieves state-of-the-art results on NeRSemble SVFR and NoW, and demonstrates strong emotion-expression capture on AffectNet, underscoring the value of neural 3DMMs for scalable monocular reconstruction. Collectively, Pix2NPHM establishes a scalable, high-quality pathway for monocular 3D facial reconstruction by combining robust geometric pretraining, diverse data, and a powerful transformer-based regressor.

Abstract

Neural Parametric Head Models (NPHMs) are a recent advancement over mesh-based 3d morphable models (3DMMs) to facilitate high-fidelity geometric detail. However, fitting NPHMs to visual inputs is notoriously challenging due to the expressive nature of their underlying latent space. To this end, we propose Pix2NPHM, a vision transformer (ViT) network that directly regresses NPHM parameters, given a single image as input. Compared to existing approaches, the neural parametric space allows our method to reconstruct more recognizable facial geometry and accurate facial expressions. For broad generalization, we exploit domain-specific ViTs as backbones, which are pretrained on geometric prediction tasks. We train Pix2NPHM on a mixture of 3D data, including a total of over 100K NPHM registrations that enable direct supervision in SDF space, and large-scale 2D video datasets, for which normal estimates serve as pseudo ground truth geometry. Pix2NPHM not only allows for 3D reconstructions at interactive frame rates, it is also possible to improve geometric fidelity by a subsequent inference-time optimization against estimated surface normals and canonical point maps. As a result, we achieve unprecedented face reconstruction quality that can run at scale on in-the-wild data.

Figures (8)

• Figure 1: Pix2NPHM is a feed-forward network that predicts NPHM giebenhain2023nphm latent codes from a single image. The latent codes can be further optimized at test-time to obtain more detailed 3D reconstructions. Here, we show mesh overlays showcasing well-aligned fittings of diverse head shapes and expressions under strong lighting conditions and occlusions. Website: https://simongiebenhain.github.io/Pix2NPHM/
• Figure 2: Motivation: Single-image 3DMM regressors are limited by their underlying 3DMM. More detailed reconstructions can be obtained by replacing FLAME FLAME with NPHM giebenhain2023nphm, and running inference-time optimization further increase fidelity (see right).
• Figure 3: Method Overview: We use pretrained ViTs, $\mathscr{E}_{\texttt{n}}$ and $\mathscr{E}_{\texttt{p}}$, as backbone, which encode the input into a token sequence. The resulting sequence is concatenated with learnable classifier token $\{\mathbf{T}^{\text{id}}_k\}$ and $\mathbf{T}^{\text{ex}}$, which are decoded into NPHM identity ($\mathbf{z}_{\text{id}}$) and expression ($\mathbf{z}_{\text{ex}}$) parameters using transformer network. We train using a 3D SDF loss, and a normal rendering loss against pseudo g.t. normals.
• Figure 4: Posed Reconstruction: We show overlays of the reconstructed meshes to judge the reconstruction alignment. Insets with a blue border depict $L_2$-Chamfer distance as an error map, rendered from a frontal camera. Red insets show the reconstructed mesh from the same camera. All our figures are best viewed digitally and zoomed-in.
• Figure 5: Neutral Reconstruction, NeRSemble: Comparison against available SotA methods on top of neutral reference image.