DNPM: A Neural Parametric Model for the Synthesis of Facial Geometric Details

TL;DR

DNPM introduces a neural parametric model that anchors high-fidelity facial geometric detail synthesis to low-dimensional semantic controls by training a StyleGAN v2 on $1024\times1024$ displacement maps from the FaceScape dataset, yielding a compact $w_+$ latent representation for details. Detailed3DMM extends traditional 3DMMs by decomposing final geometry into a proxy $V_p$ driven by identity and expression and a DNPM-informed residual $V_r$, enabling detailed detail synthesis from $\mathbf{w}_{id}$ and $\mathbf{w}_{exp}$. The approach enables two applications: speech-driven detailed 3D facial animation and 3D face reconstruction from degraded images, with extensive experiments showing improved fidelity and robustness over baselines while maintaining identity and expression consistency. Limitations include lower detail generation compared with image-to-displacement methods when high-resolution RGB inputs are available, motivating future work on larger datasets and diffusion-based generators. Overall, the work advances high-fidelity, semantically controllable facial modeling and unlocks practical tasks in animation and reconstruction.

Abstract

Parametric 3D models have enabled a wide variety of computer vision and graphics tasks, such as modeling human faces, bodies and hands. In 3D face modeling, 3DMM is the most widely used parametric model, but can't generate fine geometric details solely from identity and expression inputs. To tackle this limitation, we propose a neural parametric model named DNPM for the facial geometric details, which utilizes deep neural network to extract latent codes from facial displacement maps encoding details and wrinkles. Built upon DNPM, a novel 3DMM named Detailed3DMM is proposed, which augments traditional 3DMMs by including the synthesis of facial details only from the identity and expression inputs. Moreover, we show that DNPM and Detailed3DMM can facilitate two downstream applications: speech-driven detailed 3D facial animation and 3D face reconstruction from a degraded image. Extensive experiments have shown the usefulness of DNPM and Detailed3DMM, and the progressiveness of two proposed applications.

Figures (12)

• Figure 1: The proposed DNPM and Detailed3DMM can enable detailed 3D facial animation from driving audio, and detailed 3D face reconstruction from degraded facial image.
• Figure 2: Overview of the proposed DNPM and Detailed3DMM. The number in (b) encoder network denotes the number of channels in each layer.
• Figure 3: Visualization of the identity principal components in the form of displacement map and detailed 3D face model.
• Figure 4: Ten expression samples in the form of displacement map and detailed 3D face model.
• Figure 5: Qualitative comparisons of different encoders over FaceScape.