Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From 2D Images to 3D Model:Weakly Supervised Multi-View Face Reconstruction with Deep Fusion

Weiguang Zhao, Chaolong Yang, Jianan Ye, Rui Zhang, Yuyao Yan, Xi Yang, Bin Dong, Amir Hussain, Kaizhu Huang

TL;DR

DF-MVR addresses weakly supervised multi-view 3D face reconstruction by learning to align and fuse features across views using a dedicated mask-guided fusion backbone and a multi-layer attention module. It introduces a Face Parse Net-based mask mechanism and a photometric loss weighted by masks, along with a RedNet50-based regression and differentiable rendering to recover $3$DMM parameters under weak supervision. The approach yields RMSE improvements of $5.2\%$ on Pixel-Face and $3.0\%$ on Bosphorus over prior weakly supervised methods, without relying on dense 3D ground truth. This work demonstrates that contour-sensitive, mask-guided multi-view fusion can achieve high-precision 3D face reconstruction under weak supervision, with practical implications for scalable 3D facial modeling.

Abstract

While weakly supervised multi-view face reconstruction (MVR) is garnering increased attention, one critical issue still remains open: how to effectively interact and fuse multiple image information to reconstruct high-precision 3D models. In this regard, we propose a novel pipeline called Deep Fusion MVR (DF-MVR) to explore the feature correspondences between multi-view images and reconstruct high-precision 3D faces. Specifically, we present a novel multi-view feature fusion backbone that utilizes face masks to align features from multiple encoders and integrates one multi-layer attention mechanism to enhance feature interaction and fusion, resulting in one unified facial representation. Additionally, we develop one concise face mask mechanism that facilitates multi-view feature fusion and facial reconstruction by identifying common areas and guiding the network's focus on critical facial features (e.g., eyes, brows, nose, and mouth). Experiments on Pixel-Face and Bosphorus datasets indicate the superiority of our model. Without 3D annotation, DF-MVR achieves 5.2% and 3.0% RMSE improvement over the existing weakly supervised MVRs respectively on Pixel-Face and Bosphorus dataset. Code will be available publicly at https://github.com/weiguangzhao/DF_MVR.

From 2D Images to 3D Model:Weakly Supervised Multi-View Face Reconstruction with Deep Fusion

TL;DR

DF-MVR addresses weakly supervised multi-view 3D face reconstruction by learning to align and fuse features across views using a dedicated mask-guided fusion backbone and a multi-layer attention module. It introduces a Face Parse Net-based mask mechanism and a photometric loss weighted by masks, along with a RedNet50-based regression and differentiable rendering to recover DMM parameters under weak supervision. The approach yields RMSE improvements of on Pixel-Face and on Bosphorus over prior weakly supervised methods, without relying on dense 3D ground truth. This work demonstrates that contour-sensitive, mask-guided multi-view fusion can achieve high-precision 3D face reconstruction under weak supervision, with practical implications for scalable 3D facial modeling.

Abstract

While weakly supervised multi-view face reconstruction (MVR) is garnering increased attention, one critical issue still remains open: how to effectively interact and fuse multiple image information to reconstruct high-precision 3D models. In this regard, we propose a novel pipeline called Deep Fusion MVR (DF-MVR) to explore the feature correspondences between multi-view images and reconstruct high-precision 3D faces. Specifically, we present a novel multi-view feature fusion backbone that utilizes face masks to align features from multiple encoders and integrates one multi-layer attention mechanism to enhance feature interaction and fusion, resulting in one unified facial representation. Additionally, we develop one concise face mask mechanism that facilitates multi-view feature fusion and facial reconstruction by identifying common areas and guiding the network's focus on critical facial features (e.g., eyes, brows, nose, and mouth). Experiments on Pixel-Face and Bosphorus datasets indicate the superiority of our model. Without 3D annotation, DF-MVR achieves 5.2% and 3.0% RMSE improvement over the existing weakly supervised MVRs respectively on Pixel-Face and Bosphorus dataset. Code will be available publicly at https://github.com/weiguangzhao/DF_MVR.
Paper Structure (19 sections, 6 equations, 7 figures, 3 tables)

This paper contains 19 sections, 6 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multi-view 3D face reconstruction Methods
  4. Mask in 3D face reconstruction
  5. Main Methodology
  6. Overview
  7. Face Parse Net
  8. Deep Fusion
  9. Parametric Regression
  10. Weakly Supervised Training
  11. Photo Loss
  12. Mask Loss
  13. Overall Loss
  14. Experiment
  15. Experiment settings
  16. ...and 4 more sections

Figures (7)

  • Figure 1: Facial representation extraction with multi-view face masks and multi-layer attention mechanism. For conciseness, we do not draw the skip connection of the $\mathbf{E}_{C}$, which is similar to the $\mathbf{E}_{A}$.
  • Figure 2: Face mask annotation.
  • Figure 3: Overview of DF-MVR
  • Figure 4: Weight map
  • Figure 5: Qualitative comparison on pixel-face
  • ...and 2 more figures