Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Eye Motion Matters for 3D Face Reconstruction

Xuan Wang, Mengyuan Liu

TL;DR

The paper addresses the gap in single-image 3D face reconstruction by focusing on eye-region dynamics, which are often neglected in current methods. It introduces the Eye Landmark Adjustment Module (ELAM) and Local Dynamic Loss (LDL) to model eye-state changes and local facial dynamics within a 3DMM-based framework, using an eye-state detector and a hybrid loss to guide learning. The approach yields improved eye-state fidelity and overall 3D reconstruction accuracy, demonstrated on FFHQ, CEW, and NoW datasets, with ablations showing substantial gains from ELAM and LDL. The work advances the realism of digital humans by enabling robust closed-eye reconstructions and precise eye/mouth region modeling, with code forthcoming for public use.

Abstract

Recent advances in single-image 3D face reconstruction have shown remarkable progress in various applications. Nevertheless, prevailing techniques tend to prioritize the global facial contour and expression, often neglecting the nuanced dynamics of the eye region. In response, we introduce an Eye Landmark Adjustment Module, complemented by a Local Dynamic Loss, designed to capture the dynamic features of the eyes area. Our module allows for flexible adjustment of landmarks, resulting in accurate recreation of various eye states. In this paper, we present a comprehensive evaluation of our approach, conducting extensive experiments on two datasets. The results underscore the superior performance of our approach, highlighting its significant contributions in addressing this particular challenge.

Eye Motion Matters for 3D Face Reconstruction

TL;DR

The paper addresses the gap in single-image 3D face reconstruction by focusing on eye-region dynamics, which are often neglected in current methods. It introduces the Eye Landmark Adjustment Module (ELAM) and Local Dynamic Loss (LDL) to model eye-state changes and local facial dynamics within a 3DMM-based framework, using an eye-state detector and a hybrid loss to guide learning. The approach yields improved eye-state fidelity and overall 3D reconstruction accuracy, demonstrated on FFHQ, CEW, and NoW datasets, with ablations showing substantial gains from ELAM and LDL. The work advances the realism of digital humans by enabling robust closed-eye reconstructions and precise eye/mouth region modeling, with code forthcoming for public use.

Abstract

Recent advances in single-image 3D face reconstruction have shown remarkable progress in various applications. Nevertheless, prevailing techniques tend to prioritize the global facial contour and expression, often neglecting the nuanced dynamics of the eye region. In response, we introduce an Eye Landmark Adjustment Module, complemented by a Local Dynamic Loss, designed to capture the dynamic features of the eyes area. Our module allows for flexible adjustment of landmarks, resulting in accurate recreation of various eye states. In this paper, we present a comprehensive evaluation of our approach, conducting extensive experiments on two datasets. The results underscore the superior performance of our approach, highlighting its significant contributions in addressing this particular challenge.
Paper Structure (12 sections, 7 equations, 5 figures, 3 tables)

This paper contains 12 sections, 7 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overall Famework
  4. Basic Models
  5. ELAM: Eye Landmarks Adjustment Module
  6. LDL: Local Dynamic Loss
  7. Experiment
  8. Dataset and Settings
  9. Metrics
  10. Ablation Studies
  11. Overall Comparison
  12. Conclusion

Figures (5)

  • Figure 1: Current reconstruction problem of closed-eyes images. The photos are coming from Deep3DFaceRecon deep3d.
  • Figure 2: Overview of our 3D face reconstruction model. As highlighted in the centre, we introduced ELAM and LDL to effectively address the eye motion issue for 3D face reconstruction. The lower section of the process achieved the basic 3D mesh, while the top layer, consisting of a hybrid loss function, ensures robust face contours and skin textures.
  • Figure 3: Comparison Results on Cumulative Errors.
  • Figure 4: Cumulative Errors from NoW Benchmark.
  • Figure 5: Comparison Results on some closed-eye images.