Face Normal Estimation from Rags to Riches

TL;DR

Face normal estimation is hampered by data hunger and domain gaps between synthetic and real imagery. The authors propose a two-stage, exemplar-based framework: CP-Net learns a coarse exemplar normal from a small dataset, then NR-Net refines using both the exemplar and the input image via feature modulation; a self-attention discriminator improves the coherence of the coarse normal distribution. This design reduces data and compute requirements while delivering higher fidelity normals, as demonstrated by comprehensive experiments and ablations across multiple datasets, with the code publicly available. The work advances practical, robust face normal estimation for real-world applications by leveraging exemplar-guided refinement.

Abstract

Although recent approaches to face normal estimation have achieved promising results, their effectiveness heavily depends on large-scale paired data for training. This paper concentrates on relieving this requirement via developing a coarse-to-fine normal estimator. Concretely, our method first trains a neat model from a small dataset to produce coarse face normals that perform as guidance (called exemplars) for the following refinement. A self-attention mechanism is employed to capture long-range dependencies, thus remedying severe local artifacts left in estimated coarse facial normals. Then, a refinement network is customized for the sake of mapping input face images together with corresponding exemplars to fine-grained high-quality facial normals. Such a logical function split can significantly cut the requirement of massive paired data and computational resource. Extensive experiments and ablation studies are conducted to demonstrate the efficacy of our design and reveal its superiority over state-of-the-art methods in terms of both training expense as well as estimation quality. Our code and models are open-sourced at: https://github.com/AutoHDR/FNR2R.git.

Figures (18)

• Figure 1: Our model can generate high-quality face normals from single face images. The terms "Ex-normal" and "Re-normal" refer to the coarse exemplar normal, its refined version. "Ex-zoom" and "Re-zoom" denotes the zoomed-in exemplar and the zoomed-in refinement normal.
• Figure 2: Illustration of our exemplar-based face normal refinement framework that can generate high-quality normal by decoding the coarse exemplar and face structure features.
• Figure 3: The coarse prediction network CP-Net comparison between ours (b) and 'HFN' wang2022towards (a). Note that our coarse examplar/R is able to capture the correlation of normal direction changes where the structure changes greatly.
• Figure 4: The comparison between coarse and refined normal estimations on the FFHQ dataset karras2019style. The labeled as 'HFN-' represent the outputs generated by wang2022towards. The labels '-C' and '-R' indicate the coarse normal and refinement normal, respectively.
• Figure 5: The self-attention module in our framework operates on feature maps $x$ with dimensions $B \times C \times H \times W$, where $B$ denotes the batch size, $C$ represents the number of channels, and $H$ and $W$ denote the height and width, respectively. In the self-attention module, we set the intermediate channel size $c$ as $C/8$, and $\gamma$ is a scalar parameter that is learned during training.