SFDM: Robust Decomposition of Geometry and Reflectance for Realistic Face Rendering from Sparse-view Images

TL;DR

SFDM addresses the challenge of reconstructing photorealistic 3D faces from sparse-view images by disentangling geometry, diffuse reflectance, and specular reflectance through a two-stage, template-guided approach. The method first learns a general facial template from multi-view data and then personalizes geometry and BRDF details for each individual, incorporating subsurface scattering and an albedo gradient prior. A BRDF template with offset networks, a displacement-based geometry refinement, and a synergy mechanism between geometry and reflectance enable robust reconstruction from as few as three views and support relighting and specular editing. Experimental results on Facescape and related data show improved rendering quality and geometry accuracy over state-of-the-art methods, demonstrating SFDM’s effectiveness for realistic face synthesis under sparse views.

Abstract

In this study, we introduce a novel two-stage technique for decomposing and reconstructing facial features from sparse-view images, a task made challenging by the unique geometry and complex skin reflectance of each individual. To synthesize 3D facial models more realistically, we endeavor to decouple key facial attributes from the RGB color, including geometry, diffuse reflectance, and specular reflectance. Specifically, we design a Sparse-view Face Decomposition Model (SFDM): 1) In the first stage, we create a general facial template from a wide array of individual faces, encapsulating essential geometric and reflectance characteristics. 2) Guided by this template, we refine a specific facial model for each individual in the second stage, considering the interaction between geometry and reflectance, as well as the effects of subsurface scattering on the skin. With these advances, our method can reconstruct high-quality facial representations from as few as three images. The comprehensive evaluation and comparison reveal that our approach outperforms existing methods by effectively disentangling geometric and reflectance components, significantly enhancing the quality of synthesized novel views, and paving the way for applications in facial relighting and reflectance editing.

Figures (9)

• Figure 1: Our method learns a general facial template with distinct key facial attributes from multiple individuals. This template enables photorealistic novel-view synthesis for each individual, as well as other applications.
• Figure 2: Pipeline of SFDM. We propose a geometry and reflectance decomposition framework for 3D face reconstruction from multi-view images. We first learn a general facial template from multi-view images of multiple individuals in Stage 1. Then the learned facial template guides the further attribute refinement for each individual face model in Stage 2.
• Figure 2: Comparative evaluation metrics for ablation studies.
• Figure 3: Left: BRDF rendering, where light reflects at the incident point. Right: Subsurface scattering on facial skin, showing the offset between the incident and outgoing points.
• Figure 3: Quantitative diffuse and specular results (SSIM$\uparrow$).