Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative Face Parsing Map Guided 3D Face Reconstruction Under Occluded Scenes

Dapeng Zhao, Yue Qi

TL;DR

This work tackles single-view 3D face reconstruction under occlusion by introducing a parsing-guided synthesis pipeline. It jointly predicts 68 facial landmarks from occluded input, generates a complete face parsing map, and uses a GAN-based Face Image Synthesis Network conditioned on the parsing map to fill occluded regions before 3D estimation with a Basel Face Model and SH illumination, followed by texture refinement. The method demonstrates state-of-the-art qualitative results and improved robustness to occlusion on MICC Florence and LFW benchmarks, supported by carefully designed losses for 3D reconstruction and a strong implementation pipeline. Overall, the approach enables realistic, occlusion-robust 3D facial reconstructions with practical implications for recognition, AR, and digital avatar creation in unconstrained real-world scenes.

Abstract

Over the past few years, single-view 3D face reconstruction methods can produce beautiful 3D models. Nevertheless,the input of these works is unobstructed faces.We describe a system designed to reconstruct convincing face texture in the case of occlusion.Motivated by parsing facial features,we propose a complete face parsing map generation method guided by landmarks.We estimate the 2D face structure of the reasonable position of the occlusion area,which is used for the construction of 3D texture.An excellent anti-occlusion face reconstruction method should ensure the authenticity of the output,including the topological structure between the eyes,nose, and mouth. We extensively tested our method and its components, qualitatively demonstrating the rationality of our estimated facial structure. We conduct extensive experiments on general 3D face reconstruction tasks as concrete examples to demonstrate the method's superior regulation ability over existing methods often break down.We further provide numerous quantitative examples showing that our method advances both the quality and the robustness of 3D face reconstruction under occlusion scenes.

Generative Face Parsing Map Guided 3D Face Reconstruction Under Occluded Scenes

TL;DR

This work tackles single-view 3D face reconstruction under occlusion by introducing a parsing-guided synthesis pipeline. It jointly predicts 68 facial landmarks from occluded input, generates a complete face parsing map, and uses a GAN-based Face Image Synthesis Network conditioned on the parsing map to fill occluded regions before 3D estimation with a Basel Face Model and SH illumination, followed by texture refinement. The method demonstrates state-of-the-art qualitative results and improved robustness to occlusion on MICC Florence and LFW benchmarks, supported by carefully designed losses for 3D reconstruction and a strong implementation pipeline. Overall, the approach enables realistic, occlusion-robust 3D facial reconstructions with practical implications for recognition, AR, and digital avatar creation in unconstrained real-world scenes.

Abstract

Over the past few years, single-view 3D face reconstruction methods can produce beautiful 3D models. Nevertheless,the input of these works is unobstructed faces.We describe a system designed to reconstruct convincing face texture in the case of occlusion.Motivated by parsing facial features,we propose a complete face parsing map generation method guided by landmarks.We estimate the 2D face structure of the reasonable position of the occlusion area,which is used for the construction of 3D texture.An excellent anti-occlusion face reconstruction method should ensure the authenticity of the output,including the topological structure between the eyes,nose, and mouth. We extensively tested our method and its components, qualitatively demonstrating the rationality of our estimated facial structure. We conduct extensive experiments on general 3D face reconstruction tasks as concrete examples to demonstrate the method's superior regulation ability over existing methods often break down.We further provide numerous quantitative examples showing that our method advances both the quality and the robustness of 3D face reconstruction under occlusion scenes.

Paper Structure

This paper contains 22 sections, 16 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. Generic Face Reconstruction
  4. Face Parsing
  5. Generative Adversarial Networks
  6. Face Image Synthesis
  7. Our Approach
  8. Landmark Prediction Task
  9. Face Parsing Map Generation
  10. Face Image Synthesis with GAN
  11. Face Image Synthesis Network
  12. Loss Function
  13. Camera and Illumination Model
  14. Loss Function of 3D Reconstruction
  15. Implementation Details
  16. ...and 7 more sections

Figures (5)

  • Figure 1: Overall our pipeline. We first remove the occluded area and reconstruct the face with complete facial features. Then we utilize ResNet-50 and texture refinement network to reconstruct the final 3D model.
  • Figure 2: Our face parsing map generation module, which follows Algorithm \ref{['suanfa:001']}. The results shown in the figure show that our method finally successfully removed the occlusion of fingers and hair
  • Figure 3: Comparison of qualitative results. Baseline methods from left to right: 3DDFA, $\text{D}{{\text{F}}^{\text{2}}}\text{Net}$, Chen et al. and our method.
  • Figure 4: Comparison of error heat maps on the MICC Florence datasets. Digits denote $90\%$ error (mm).
  • Figure 5: Reconstructions with occlusions. Left: Qualitative results of Sela et al.RN200 and our shape. Right: LFW verification ROC for the shapes, with and without occlusions.