3D Face Reconstruction With Geometry Details From a Single Color Image Under Occluded Scenes

TL;DR

This work tackles single-image 3D face reconstruction under occlusions by introducing a bump-map based approach that adds geometry details to a robust base shape. It presents a two-stage pipeline: first synthesize an occlusion-free 2D face using a face parsing map and a face edge lines map, then recover dense 3D geometry by augmenting the base shape with a learnable bump map and regressing subject- and expression-specific coefficients, aided by spherical-harmonics lighting. Key contributions include a novel Face Image Synthesis Network for eyeglasses removal, an enhanced occlusion-aware loss for geometry, and a unified framework that demonstrates state-of-the-art qualitative performance on real-world occluded images. The approach improves the reliability and realism of 3D facial geometry in occluded scenarios, enabling applications in digital animation, video editing, and recognition where occluders are present.

Abstract

3D face reconstruction technology aims to generate a face stereo model naturally and realistically. Previous deep face reconstruction approaches are typically designed to generate convincing textures and cannot generalize well to multiple occluded scenarios simultaneously. By introducing bump mapping, we successfully added mid-level details to coarse 3D faces. More innovatively, our method takes into account occlusion scenarios. Thus on top of common 3D face reconstruction approaches, we in this paper propose a unified framework to handle multiple types of obstruction simultaneously (e.g., hair, palms and glasses et al.).Extensive experiments and comparisons demonstrate that our method can generate high-quality reconstruction results with geometry details from captured facial images under occluded scenes.

Figures (5)

• Figure 1: Method overview. At first, as input for our face image synthesis network, we need the target image ${{\mathbf{I}}_{\mathbf{in}}}$ and map ${{\mathbf{M}}_{\mathbf{fin}}}$. We utilize the face parsing map generation module and edge lines map generation module to obtain the map ${{\mathbf{M}}_{\mathbf{fa}}}$ and ${{\mathbf{M}}_{\mathbf{edge}}}$. Then we obtain the final face parsing map ${{\mathbf{M}}_{\mathbf{fin}}}$ following Zhao et al.'s AlgorithmRN852. After obtaining the face image ${{\mathbf{I}}_{\mathbf{out}}}$ with eyeglasses removed, in step two, we leverage ResNet-50 and texture refinement network to reconstruct the final 3D model.
• Figure 2: The overview of the proposed face parsing network.
• Figure 3: The overview of the proposed face edge lines map generation approach.
• Figure 4: Comparison of qualitative results. Baseline methods from left to right: Sela et al., PRNet,3DDFA and our method.
• Figure 5: Reconstructions with eyeglasses. Left: Qualitative results of Sela et al.RN200 and our shape. Right: LFW verification ROC for the shapes, with and without eyeglasses.