Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative Landmarks Guided Eyeglasses Removal 3D Face Reconstruction

Dapeng Zhao, Yue Qi

TL;DR

This work tackles single-view 3D face reconstruction under eyeglasses occlusion by jointly learning landmark estimation, face-parsing guided eyeglasses removal, and a GAN-based face synthesis module to produce an eyeglasses-free texture. A Deleter-GAN architecture removes glasses using face-parsing maps, while a Generator infills the occluded regions using calibrated landmarks, guided by a Discriminator with pixel, style, and variation losses. The 3D shape is then recovered via a 3DMM framework with perspective projection and SH lighting, optimized with pixel-wise and FaceNet-based feature losses. Experiments on MICC Florence and LFW demonstrate state-of-the-art qualitative and quantitative performance, showing improved robustness to eyeglasses and better preservation of authentic facial topology in-the-wild.

Abstract

Single-view 3D face reconstruction is a fundamental Computer Vision problem of extraordinary difficulty. Current systems often assume the input is unobstructed faces which makes their method not suitable for in-the-wild conditions. We present a method for performing a 3D face that removes eyeglasses from a single image. Existing facial reconstruction methods fail to remove eyeglasses automatically for generating a photo-realistic 3D face "in-the-wild".The innovation of our method lies in a process for identifying the eyeglasses area robustly and remove it intelligently. In this work, we estimate the 2D face structure of the reasonable position of the eyeglasses area, which is used for the construction of 3D texture. An excellent anti-eyeglasses face reconstruction method should ensure the authenticity of the output, including the topological structure between the eyes, nose, and mouth. We achieve this via a deep learning architecture that performs direct regression of a 3DMM representation of the 3D facial geometry from a single 2D image. We also demonstrate how the related face parsing task can be incorporated into the proposed framework and help improve reconstruction quality. We conduct extensive experiments on existing 3D face reconstruction tasks as concrete examples to demonstrate the method's superior regulation ability over existing methods often break down.

Generative Landmarks Guided Eyeglasses Removal 3D Face Reconstruction

TL;DR

This work tackles single-view 3D face reconstruction under eyeglasses occlusion by jointly learning landmark estimation, face-parsing guided eyeglasses removal, and a GAN-based face synthesis module to produce an eyeglasses-free texture. A Deleter-GAN architecture removes glasses using face-parsing maps, while a Generator infills the occluded regions using calibrated landmarks, guided by a Discriminator with pixel, style, and variation losses. The 3D shape is then recovered via a 3DMM framework with perspective projection and SH lighting, optimized with pixel-wise and FaceNet-based feature losses. Experiments on MICC Florence and LFW demonstrate state-of-the-art qualitative and quantitative performance, showing improved robustness to eyeglasses and better preservation of authentic facial topology in-the-wild.

Abstract

Single-view 3D face reconstruction is a fundamental Computer Vision problem of extraordinary difficulty. Current systems often assume the input is unobstructed faces which makes their method not suitable for in-the-wild conditions. We present a method for performing a 3D face that removes eyeglasses from a single image. Existing facial reconstruction methods fail to remove eyeglasses automatically for generating a photo-realistic 3D face "in-the-wild".The innovation of our method lies in a process for identifying the eyeglasses area robustly and remove it intelligently. In this work, we estimate the 2D face structure of the reasonable position of the eyeglasses area, which is used for the construction of 3D texture. An excellent anti-eyeglasses face reconstruction method should ensure the authenticity of the output, including the topological structure between the eyes, nose, and mouth. We achieve this via a deep learning architecture that performs direct regression of a 3DMM representation of the 3D facial geometry from a single 2D image. We also demonstrate how the related face parsing task can be incorporated into the proposed framework and help improve reconstruction quality. We conduct extensive experiments on existing 3D face reconstruction tasks as concrete examples to demonstrate the method's superior regulation ability over existing methods often break down.
Paper Structure (22 sections, 9 equations, 6 figures, 1 table)

This paper contains 22 sections, 9 equations, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generic Face Reconstruction
  4. Face Parsing
  5. Deep Face Synthesis Methods
  6. Our Method
  7. Landmark Estimation Network
  8. Face Synthesis Module
  9. Deleter.
  10. Generator.
  11. Discriminator.
  12. 3D Face Reconstruction
  13. Pixel-wise Loss.
  14. Face Features Loss.
  15. Experimental Details and Results
  16. ...and 7 more sections

Figures (6)

  • Figure 1: Our method overview. See related sections for details.
  • Figure 2: Landmark prediction network of our method.
  • Figure 3: Face parsing module of our method.
  • Figure 4: Comparison of qualitative results. Baseline methods from left to right: 3DDFA, DF2Net, Chen et al.,PRNet, and our method.
  • Figure 5: Comparison of error heat maps on the MICC Florence datasets. Digits denote $90$% error (mm).
  • ...and 1 more figures