ImFace++: A Sophisticated Nonlinear 3D Morphable Face Model with Implicit Neural Representations

TL;DR

Comprehensive qualitative and quantitative evaluation demonstrates that ImFace++ significantly advances the state-of-the-art in terms of both face reconstruction fidelity and correspondence accuracy.

Abstract

Accurate representations of 3D faces are of paramount importance in various computer vision and graphics applications. However, the challenges persist due to the limitations imposed by data discretization and model linearity, which hinder the precise capture of identity and expression clues in current studies. This paper presents a novel 3D morphable face model, named ImFace++, to learn a sophisticated and continuous space with implicit neural representations. ImFace++ first constructs two explicitly disentangled deformation fields to model complex shapes associated with identities and expressions, respectively, which simultaneously facilitate automatic learning of point-to-point correspondences across diverse facial shapes. To capture more sophisticated facial details, a refinement displacement field within the template space is further incorporated, enabling fine-grained learning of individual-specific facial details. Furthermore, a Neural Blend-Field is designed to reinforce the representation capabilities through adaptive blending of an array of local fields. In addition to ImFace++, we devise an improved learning strategy to extend expression embeddings, allowing for a broader range of expression variations. Comprehensive qualitative and quantitative evaluation demonstrates that ImFace++ significantly advances the state-of-the-art in terms of both face reconstruction fidelity and correspondence accuracy.

Figures (20)

• Figure 1: ImFace++ encodes intricate variations associated with identities and expressions by two explicitly disentangled deformation fields with respect to a template face. Leveraging the established point-to-point correspondence, it applies an additional refinement displacement field to capture high-frequency details in the individual-independent template space. The model finally yields sophisticated and implicit representations that are highly adaptable for 3D face modeling.
• Figure 2: ImFace++ overview. The model is constructed through two-stage learning. (a) The first stage primarily focuses on nonlinear deformation modeling and surface correspondence between different faces. The network comprises three Mini-Nets blocks designed to explicitly disentangle shape morphs into separate deformation fields, where the Expression and Identity Mini-Nets blocks are associated with expression and identity deformations, respectively. Meanwhile, the Template Mini-Nets block learns the SDF of a template face shape. (b) With the point-to-point correspondence established, ImFace++ proceeds to employ a Detail Mini-Nets block in the second stage, responsible for learning refinement displacement field that encodes high-frequency geometry details.
• Figure 3: Neural Blend-Field.(a) The Mini-Nets block is a shared architecture, which decomposes an entire facial feature into semantically meaningful parts and encodes them by a set of local field functions. It is tailed by a Fusion Network for more comprehensive representations. (b) The Landmark-Net is introduced to softly partition the entire facial surface. (c) The Fusion Network is a lightweight module conditioned on the query point position, which adaptively blends the local field functions, resulting in an elaborate Neural Blend-Field.
• Figure 4: (a) SDF is able to represent closed shapes. (b) UDF is capable of representing an open surface but the gradient is discontinuous at the boundary, making it hard to be fitted by neural networks. (c) The proposed method generates pseudo watertight faces and restricts implicit functions on them, enabling implicit neural networks to learn geometry representations on 3D faces.
• Figure 5: Illustration of the preprocessing pipeline. The original face data from FaceScape yang2020facescape undergo initial normalization and cropping. Hidden surfaces are subsequently removed. The Delaunay Triangulation algorithm lee1980two is then employed to achieve watertight results. Lastly, we sample points and estimate the SDF and normals for training ImFace++.