FaceRefiner: High-Fidelity Facial Texture Refinement with Differentiable Rendering-based Style Transfer

TL;DR

This paper proposes a style transfer-based facial texture refinement method named FaceRefiner, which can improve the texture quality and the face identity preserving ability, compared with state-of-the-arts methods.

Abstract

Recent facial texture generation methods prefer to use deep networks to synthesize image content and then fill in the UV map, thus generating a compelling full texture from a single image. Nevertheless, the synthesized texture UV map usually comes from a space constructed by the training data or the 2D face generator, which limits the methods' generalization ability for in-the-wild input images. Consequently, their facial details, structures and identity may not be consistent with the input. In this paper, we address this issue by proposing a style transfer-based facial texture refinement method named FaceRefiner. FaceRefiner treats the 3D sampled texture as style and the output of a texture generation method as content. The photo-realistic style is then expected to be transferred from the style image to the content image. Different from current style transfer methods that only transfer high and middle level information to the result, our style transfer method integrates differentiable rendering to also transfer low level (or pixel level) information in the visible face regions. The main benefit of such multi-level information transfer is that, the details, structures and semantics in the input can thus be well preserved. The extensive experiments on Multi-PIE, CelebA and FFHQ datasets demonstrate that our refinement method can improve the texture quality and the face identity preserving ability, compared with state-of-the-arts.

Figures (16)

• Figure 1: The refined facial textures by our proposed FaceRefiner on the the results produced by OSTEC gecer2021ostec, can yield more eye features (1st row) and skin spots (2nd row), and thus better preserve the face identity.
• Figure 2: The overview of our proposed FaceRefiner. The inputs of FaceRefiner include the face image $I$, the 3D face reconstruction results (3D model $\mathcal{M}$ and camera pose $\mathcal{P}$, sampled texture $I_S$) and the initial imperfect texture $I_C$ produced by an existing facial texture generation method. The differentiable rendering-based style transfer is adopted to improve the quality of $I_C$. The differentiable renderer is employed to produce rendered image $I_R$ of the inputted camera pose $\mathcal{P}$. Then the rendering loss is calculated to measure the inconsistency between rendered and inputted image, and the gradients are back-propagated to a classical style transfer module containing style and content loss to optimize the facial texture $I_X$.
• Figure 3: The generation of style image.
• Figure 4: In 1st row, the style and content images are generated in the proposed way. In 2nd row, the style and content images are swapped.
• Figure 5: The illustration of hypercolumn matching between style image and output image. (c)-(e) show the output images produced by performing different stages of optimization.