3D Face Reconstruction Error Decomposed: A Modular Benchmark for Fair and Fast Method Evaluation

TL;DR

This work introduces M3DFB, a modular benchmarking toolkit that decouples the steps involved in computing geometric error for 3D face reconstruction, enabling fair, fast, and extensible evaluation. A novel correction step (ETC) and the use of non-rigid warping with landmarks significantly improve error estimation accuracy while reducing computation time, challenging the ubiquity of ICP-based benchmarks. Across synthetic and real datasets with BFM and FLAME topologies, ELR-based methods coupled with ETC achieve accuracy comparable to top NICP-based estimators but at an order of magnitude faster, underscoring the importance of modular benchmarking for robust method comparison and training of learned reconstructions. The open-source framework supports easy integration of new estimators and experiments, promoting reproducibility and more reliable progress in 3D face reconstruction benchmarking and learning-based methods.

Abstract

Computing the standard benchmark metric for 3D face reconstruction, namely geometric error, requires a number of steps, such as mesh cropping, rigid alignment, or point correspondence. Current benchmark tools are monolithic (they implement a specific combination of these steps), even though there is no consensus on the best way to measure error. We present a toolkit for a Modularized 3D Face reconstruction Benchmark (M3DFB), where the fundamental components of error computation are segregated and interchangeable, allowing one to quantify the effect of each. Furthermore, we propose a new component, namely correction, and present a computationally efficient approach that penalizes for mesh topology inconsistency. Using this toolkit, we test 16 error estimators with 10 reconstruction methods on two real and two synthetic datasets. Critically, the widely used ICP-based estimator provides the worst benchmarking performance, as it significantly alters the true ranking of the top-5 reconstruction methods. Notably, the correlation of ICP with the true error can be as low as 0.41. Moreover, non-rigid alignment leads to significant improvement (correlation larger than 0.90), highlighting the importance of annotating 3D landmarks on datasets. Finally, the proposed correction scheme, together with non-rigid warping, leads to an accuracy on a par with the best non-rigid ICP-based estimators, but runs an order of magnitude faster. Our open-source codebase is designed for researchers to easily compare alternatives for each component, thus helping accelerating progress in benchmarking for 3D face reconstruction and, furthermore, supporting the improvement of learned reconstruction methods, which depend on accurate error estimation for effective training.

Figures (13)

• Figure 1: The main steps involved in the computation of the geometric error between reconstructed meshes ($R$) and ground truth scans ($G$). Note that mesh cropping, non-rigid warping, and correction are optional steps. The methods that our toolkit implements for each step are listed next to a bullet point ($\cdot$)
• Figure 2: (a) Incorrect Chamfer (i.e., nearest-neighbor) correspondences (Ch.) and the improvement achieved by non-rigid warping via ELR, illustrated by overlaying (red) landmark points from the reconstructed mesh $R$ that correspond to eyes, brows, mouth and nose on corresponding ground truth scans. (b) Landmark points used as reference for rigid- and non-rigid alignment.
• Figure 3: Re-meshing: blue points belong to $G$, re-meshed (red) points $\check{g}_i$ are obtained as barycenters of facets of $G$
• Figure 4: Illustration of ETC. Incorrect Chamfer-based point correspondences between $R$ and $G$ often lead to visible gaps and fractures in the matched mesh $\hat{G}$. The correction term $\Delta \hat{G}$ exploits this by penalizing for such gaps.
• Figure 5: Rate of inconsistency of compared estimators on BFM and FLAME datasets, and the average of the two rates (the lower the better).