Towards Geometric-Photometric Joint Alignment for Facial Mesh Registration

TL;DR

This paper tackles the challenge of registering facial meshes across diverse expressions with dense photometric consistency and topology preservation. It introduces GPJA, a differentiable-rendering-based framework that jointly optimizes geometry and texture through Holistic Rendering Alignment and a multiscale regularized optimization, without requiring semantic annotations or pre-aligned training data. Key contributions include the HRA constraints (color, depth, normals), a robust coarse-to-fine optimization scheme, and the creation of a subject-specific textured template mesh with a shared texture space for all expressions. Experimental results show GPJA achieving pixel-level geometric and photometric alignment, outperforming non-rigid ICP and state-of-the-art deep-learning approaches, and enabling batch re-parametrization and animation workflows with consistent textures across expressions.

Abstract

This paper presents a Geometric-Photometric Joint Alignment~(GPJA) method, which aligns discrete human expressions at pixel-level accuracy by combining geometric and photometric information. Common practices for registering human heads typically involve aligning landmarks with facial template meshes using geometry processing approaches, but often overlook dense pixel-level photometric consistency. This oversight leads to inconsistent texture parametrization across different expressions, hindering the creation of topologically consistent head meshes widely used in movies and games. GPJA overcomes this limitation by leveraging differentiable rendering to align vertices with target expressions, achieving joint alignment in both geometry and photometric appearances automatically, without requiring semantic annotation or pre-aligned meshes for training. It features a holistic rendering alignment mechanism and a multiscale regularized optimization for robust convergence on large deformation. The method utilizes derivatives at vertex positions for supervision and employs a gradient-based algorithm which guarantees smoothness and avoids topological artifacts during the geometry evolution. Experimental results demonstrate faithful alignment under various expressions, surpassing the conventional non-rigid ICP-based methods and the state-of-the-art deep learning based method. In practical, our method generates meshes of the same subject across diverse expressions, all with the same texture parametrization. This consistency benefits face animation, re-parametrization, and other batch operations for face modeling and applications with enhanced efficiency.

Figures (14)

• Figure 1: Given multiview images and (a) the textured template mesh, we propose a novel method GPJA based on differentiable rendering to achieve geometric and photometric alignment jointly for facial meshes. (b) The aligned meshes are rendered with the shared texture map as the template in (a). (c) The zoomed renderings (yellow boxes) of eyes and mouths demonstrate photometric alignment with the reference images (black boxes).
• Figure 2: Illustration of our proposed GPJA. With the provided textured template, scans and camera poses, the constraints from HRA $\mathcal{L}$ back-propagates derivatives at each iteration to guide the warping of the template. The regularized optimization is built on a multiscale scheme with periodic tessellation, and the vertices are updated with a robust modified gradient descent algorithm.
• Figure 3: Errors are indicated with arrows: (a) Landmark SPIGA errors under asymmetry, occlusion, and extreme expressions, and (b) Optical flow liu2009beyond inaccuracies due to significant visibility changes.
• Figure 4: Illustration of the masking strategy. Using the same parametrization as (a) the texture map, the interior mouth is manually masked out, resulting in (b) a binary mask image. By the shading operation $F_{S}(\boldsymbol{x}|\boldsymbol{P}_{j},\boldsymbol{B})$, (c) the mouth socket is mask out from the color constraint.
• Figure 5: The pipeline of the textured template mesh creation. (a) A genus-0 mesh in the shape of a bust that approximately overlaps the scan undergoes GPJA as Fig. \ref{['fig:mainpipeline']}, producing (b) a densely tessellated mesh accompanied with the reconstructed texture map $\boldsymbol{C}_{\boldsymbol{T}}$, which is then decimated to construct (c) a coarse template $\boldsymbol{T}$, while still preserving the same texture map.