Pixel3DMM: Versatile Screen-Space Priors for Single-Image 3D Face Reconstruction

TL;DR

Pixel3DMM presents a pair of vision transformers that predict per-pixel surface normals and uv-coordinates to constrain FLAME-based single-image 3D face reconstruction, leveraging foundation-model features and training on large, FLAME-aligned datasets. The method yields a robust optimization-based reconstruction with a novel 2D-vertex loss that translates pixel-space cues into 3D constraints, achieving state-of-the-art results on posed geometry and competitive neutral reconstructions. A new NeRSemble-derived benchmark enables simultaneous evaluation of posed and neutral geometry, highlighting strengths in disambiguating identity and expression. The work demonstrates strong performance for single-image 3D face reconstruction, introduces a scalable training protocol, and provides a public benchmark to advance future research in this area.

Abstract

We address the 3D reconstruction of human faces from a single RGB image. To this end, we propose Pixel3DMM, a set of highly-generalized vision transformers which predict per-pixel geometric cues in order to constrain the optimization of a 3D morphable face model (3DMM). We exploit the latent features of the DINO foundation model, and introduce a tailored surface normal and uv-coordinate prediction head. We train our model by registering three high-quality 3D face datasets against the FLAME mesh topology, which results in a total of over 1,000 identities and 976K images. For 3D face reconstruction, we propose a FLAME fitting opitmization that solves for the 3DMM parameters from the uv-coordinate and normal estimates. To evaluate our method, we introduce a new benchmark for single-image face reconstruction, which features high diversity facial expressions, viewing angles, and ethnicities. Crucially, our benchmark is the first to evaluate both posed and neutral facial geometry. Ultimately, our method outperforms the most competitive baselines by over 15% in terms of geometric accuracy for posed facial expressions.

Figures (6)

• Figure 1: We present Pixel3DMM, a set of two ViTs dosovitskiy2020vit, which are tailored to predict per-pixel surface normals and uv-coordinates. Here, we demonstrate the fidelity and robustness of our prior networks on examples from the FFHQ karras2019ffhq dataset. From top to bottom we show input RGB, predicted surface normals, 2D vertices extracted from the uv-coordinate prediction, and our FLAME fitting results.
• Figure 2: Method Overview: Pixel3DMM consists of (a) learning pixel-aligned geometric priors (left) and (b) test-time optimization against predicted uv-coordinates and normals (right). On the left we illustrate our network architecture and examples from the training set. On the right we illustrate to process of finding per-vertex 2D locations using a nearest neighbor (N.N.) look up, and our loss terms.
• Figure 3: 3D Face Reconstruction Benchmark Analysis. We show the 5 most diverse images from each benchmark dataset, as measured by the expression codes of EMOCA danvevcek2022emoca. Our benchmark covers a richer diversity of facial expressions.
• Figure 4: Qualitative Comparison (Posed): We show overlays of the reconstructed meshes to judge the reconstruction alignment. Insets with a blue border depict $L_2$-Chamfer distance as an error map, rendered from a frontal camera. Red insets show the reconstructed mesh from the same camera. We encourage the reviewers to watch our supplementary material for additional visualizations.
• Figure 5: Qualitative Comparison (Neutral): Alignment of the neutral prediction against the neutral image and scan of a person.