Geometry-Aware Face Reconstruction Under Occluded Scenes
Dapeng Zhao
TL;DR
Occluded scenes severely challenge single-image 3D face reconstruction. The authors propose a two-stage geometry-aware pipeline that first removes occlusions guided by a face parsing map and landmark predictions, then fits a Basel Face Model representation and refines depth with a bump-map displacement. Key contributions include the integration of landmark prediction with parsing-guided occlusion removal, a GAN-based synthesis module with per-pixel and style losses, and a geometry-refinement loss $L_{geo}$ that preserves high-frequency details via depth displacement $d'(b)=d(b)+\phi^{-1}(\Phi(b))$. Empirical results demonstrate improved geometric fidelity in occluded regions and robustness against occlusion leakage compared to several baselines, indicating strong practical value for unconstrained face analysis.
Abstract
Recently, deep learning-based 3D face reconstruction methods have demonstrated promising advancements in terms of quality and efficiency. Nevertheless, these techniques face challenges in effectively handling occluded scenes and fail to capture intricate geometric facial details. Inspired by the principles of GANs and bump mapping, we have successfully addressed these issues. Our approach aims to deliver comprehensive 3D facial reconstructions, even in the presence of occlusions.While maintaining the overall shape's robustness, we introduce a mid-level shape refinement to the fundamental structure. Furthermore, we illustrate how our method adeptly extends to generate plausible details for obscured facial regions. We offer numerous examples that showcase the effectiveness of our framework in producing realistic results, where traditional methods often struggle. To substantiate the superior adaptability of our approach, we have conducted extensive experiments in the context of general 3D face reconstruction tasks, serving as concrete evidence of its regulatory prowess compared to manual occlusion removal methods.