GRAPE: Generalizable and Robust Multi-view Facial Capture

TL;DR

GRAPE tackles the generalization problem in multi-view facial capture across camera arrays by introducing a camera-array-agnostic initialization based on a visual hull and a visibility-aware 3D feature aggregation. It employs a coarse-to-fine vertex prediction pipeline and a novel update-by-disagreement training strategy to robustly handle registration and scan noise. The method achieves competitive accuracy on FaMoS and superior generalization on FaceScape compared with TEMPEH and baselines, with additional gains from finetuning on target data. This approach significantly reduces the data collection and processing burden when deploying capture systems on new camera arrays, enabling more flexible and robust facial performance capture in practice.

Abstract

Deep learning-based multi-view facial capture methods have shown impressive accuracy while being several orders of magnitude faster than a traditional mesh registration pipeline. However, the existing systems (e.g. TEMPEH) are strictly restricted to inference on the data captured by the same camera array used to capture their training data. In this study, we aim to improve the generalization ability so that a trained model can be readily used for inference (i.e. capture new data) on a different camera array. To this end, we propose a more generalizable initialization module to extract the camera array-agnostic 3D feature, including a visual hull-based head localization and a visibility-aware 3D feature aggregation module enabled by the visual hull. In addition, we propose an ``update-by-disagreement'' learning strategy to better handle data noise (e.g. inaccurate registration, scan noise) by discarding potentially inaccurate supervision signals during training. The resultant generalizable and robust topologically consistent multi-view facial capture system (GRAPE) can be readily used to capture data on a different camera array, reducing great effort on data collection and processing. Experiments on the FaMoS and FaceScape datasets demonstrate the effectiveness of the proposed method.

Figures (6)

• Figure 1: Overview of GRAPE. GRAPE consists of a non-data-driven and thus generalizable initialization module, and two 3D ConvNets learned in a coarse-to-fine manner to predict the mesh vertex locations. (1) The initialization module first calculates a visual hull from the foreground masks and then aggregates 2D image features with the consideration of visibility, resulting in a camera setup-agnostic 3D feature cube generalizable to different camera arrays. (2) The networks are trained with an "update-by-disagreement" learning strategy to handle registration errors and scan noises, where two networks are used in each stage to discard unreliable supervision signals.
• Figure 2: Per-voxel visibility check. The provided input view is visible for all the yellow voxels. A voxel aggregates image features only from visible views by aggregating 2D image features from its projected location.
• Figure 3: Qualitative comparisons on the FaMoS dataset. For each model, we show the predicted mesh (left) and the raw scan mesh (right) colored with the point-to-surface distance. The corresponding color bar (3 mm max.) is shown on the right.
• Figure 4: Qualitative comparisons on the FaceScape dataset. For each model, we show the predicted mesh (left) and the raw scan mesh (right) colored with the point-to-surface distance. The corresponding color bar (3 mm max.) is shown on the right.
• Figure 5: Qualitative comparison of generalization - FaMoS $\rightarrow$ FaceScape. The models are trained on the FaMoS data and evaluated on the FaceScape dataset. For each model, we show the predicted mesh (left) and the raw scan mesh (right) colored with the point-to-surface distance. The corresponding color bar (3 mm max.) is shown on the right.