FaceCom: Towards High-fidelity 3D Facial Shape Completion via Optimization and Inpainting Guidance

TL;DR

FaceCom tackles 3D facial shape completion for incomplete scans by learning a mesh-based generator via a graph convolutional autoencoder whose latent vector $oldsymbol{z}$ is constrained to the hypersphere using a regularizer $L_{reg}$; an optimization step fits $oldsymbol{z}$ and a rigid transform by minimizing $L_{fit}$, guided by image inpainting through $L_{inp}$, and a post-processing stage refines the result. The method operates directly on mesh data, avoiding strict vertex correspondences, and uses a differentiable renderer and PyTorch3D for optimization, enabling input flexibility from meshes, point clouds, or keypoints. Training leverages a mixed 3D facial dataset totaling 2405 identities, enabling robust fusion of global and local geometry and sampling from the hypersphere for diverse generation. Empirical results across fitting, shape completion, clinical prosthetics, and non-rigid registration show FaceCom achieving superior fidelity and practical utility, outperforming several state-of-the-art baselines and providing a complete pipeline that can assist facial prosthetics design and clinical data processing.

Abstract

We propose FaceCom, a method for 3D facial shape completion, which delivers high-fidelity results for incomplete facial inputs of arbitrary forms. Unlike end-to-end shape completion methods based on point clouds or voxels, our approach relies on a mesh-based generative network that is easy to optimize, enabling it to handle shape completion for irregular facial scans. We first train a shape generator on a mixed 3D facial dataset containing 2405 identities. Based on the incomplete facial input, we fit complete faces using an optimization approach under image inpainting guidance. The completion results are refined through a post-processing step. FaceCom demonstrates the ability to effectively and naturally complete facial scan data with varying missing regions and degrees of missing areas. Our method can be used in medical prosthetic fabrication and the registration of deficient scanning data. Our experimental results demonstrate that FaceCom achieves exceptional performance in fitting and shape completion tasks. The code is available at https://github.com/dragonylee/FaceCom.git.

Figures (7)

• Figure 1: FaceCom Overview. During the shape fitting stage, the shape generator is employed to fit the incomplete facial scan $M_{d}$ using an optimization method. The optimization objective comprises several loss functions, including the disparity between the input $M_{d}$ and the fitting result $M_{fit}$, the difference between the images $I_{fit}$ and $I_{inp}$, and regularization terms. Following the acquisition of the optimal complete shape relative to the incomplete input, a post-processing step is applied to achieve the final completion results $M_{out}$.
• Figure 2: Architecture of shape generator. Left: Network used for training, which is primarily based on a graph convolutional autoencoder structure. It conducts feature extraction and generation separately from both local and global perspectives. After training, the decoder component serves as the shape generator for FaceCom. Right: Our shape generator can produce diverse facial samples from the hypersphere space and is readily optimized within the latent space.
• Figure 3: Visualization of post-processing technique. \ref{['fig:post-a']} depicts the incomplete facial scan input, \ref{['fig:post-b']} demonstrates the fitting result, \ref{['fig:post-c']} to \ref{['fig:post-e']} illustrate each step of the post-processing process. \ref{['fig:post-f']} and \ref{['fig:post-g']} showcase the disparities between the results before and after post-processing and the incomplete input.
• Figure 4: Fitting experiment examples.
• Figure 5: Examples of facial shape completion experiment. FaceCom demonstrates high-quality completion results for facial inputs with defects of various positions and sizes, even for samples with numerous wrinkles or lower quality. Please refer to our supplementary materials for more detailed views.