Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Supervision-by-Hallucination-and-Transfer: A Weakly-Supervised Approach for Robust and Precise Facial Landmark Detection

Jun Wan, Yuanzhi Yao, Zhihui Lai, Jie Zhou, Xianxu Hou, Wenwen Min

TL;DR

This paper tackles the challenge of precise facial landmark detection on low-resolution inputs with limited annotations by introducing a weakly-supervised framework, SHT, that couples a Dual Hallucination Learning Network (DHLN) with a Facial Pose Transfer Network (FPTN). DHLN jointly learns landmark heatmap hallucination and high-resolution face hallucination, while FPTN refines these outputs through pose transfer, with FPTN unused at inference to avoid extra cost. The two components are trained in a mutually beneficial loop, and unlabeled high-resolution data are leveraged to boost both FLD and face hallucination via a novel loss formulation that includes a truncated DHLN term for unlabeled data. Experiments on CelebA, Helen, 300W, AFLW, WFLW and 300VW demonstrate state-of-the-art performance and robust improvements, highlighting the practical impact of integrating hallucination and pose transfer under weak supervision for low-resolution facial analysis.

Abstract

High-precision facial landmark detection (FLD) relies on high-resolution deep feature representations. However, low-resolution face images or the compression (via pooling or strided convolution) of originally high-resolution images hinder the learning of such features, thereby reducing FLD accuracy. Moreover, insufficient training data and imprecise annotations further degrade performance. To address these challenges, we propose a weakly-supervised framework called Supervision-by-Hallucination-and-Transfer (SHT) for more robust and precise FLD. SHT contains two novel mutually enhanced modules: Dual Hallucination Learning Network (DHLN) and Facial Pose Transfer Network (FPTN). By incorporating FLD and face hallucination tasks, DHLN is able to learn high-resolution representations with low-resolution inputs for recovering both facial structures and local details and generating more effective landmark heatmaps. Then, by transforming faces from one pose to another, FPTN can further improve landmark heatmaps and faces hallucinated by DHLN for detecting more accurate landmarks. To the best of our knowledge, this is the first study to explore weakly-supervised FLD by integrating face hallucination and facial pose transfer tasks. Experimental results of both face hallucination and FLD demonstrate that our method surpasses state-of-the-art techniques.

Supervision-by-Hallucination-and-Transfer: A Weakly-Supervised Approach for Robust and Precise Facial Landmark Detection

TL;DR

This paper tackles the challenge of precise facial landmark detection on low-resolution inputs with limited annotations by introducing a weakly-supervised framework, SHT, that couples a Dual Hallucination Learning Network (DHLN) with a Facial Pose Transfer Network (FPTN). DHLN jointly learns landmark heatmap hallucination and high-resolution face hallucination, while FPTN refines these outputs through pose transfer, with FPTN unused at inference to avoid extra cost. The two components are trained in a mutually beneficial loop, and unlabeled high-resolution data are leveraged to boost both FLD and face hallucination via a novel loss formulation that includes a truncated DHLN term for unlabeled data. Experiments on CelebA, Helen, 300W, AFLW, WFLW and 300VW demonstrate state-of-the-art performance and robust improvements, highlighting the practical impact of integrating hallucination and pose transfer under weak supervision for low-resolution facial analysis.

Abstract

High-precision facial landmark detection (FLD) relies on high-resolution deep feature representations. However, low-resolution face images or the compression (via pooling or strided convolution) of originally high-resolution images hinder the learning of such features, thereby reducing FLD accuracy. Moreover, insufficient training data and imprecise annotations further degrade performance. To address these challenges, we propose a weakly-supervised framework called Supervision-by-Hallucination-and-Transfer (SHT) for more robust and precise FLD. SHT contains two novel mutually enhanced modules: Dual Hallucination Learning Network (DHLN) and Facial Pose Transfer Network (FPTN). By incorporating FLD and face hallucination tasks, DHLN is able to learn high-resolution representations with low-resolution inputs for recovering both facial structures and local details and generating more effective landmark heatmaps. Then, by transforming faces from one pose to another, FPTN can further improve landmark heatmaps and faces hallucinated by DHLN for detecting more accurate landmarks. To the best of our knowledge, this is the first study to explore weakly-supervised FLD by integrating face hallucination and facial pose transfer tasks. Experimental results of both face hallucination and FLD demonstrate that our method surpasses state-of-the-art techniques.
Paper Structure (14 sections, 10 equations, 7 figures, 6 tables)

This paper contains 14 sections, 10 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Supervision-by-Hallucination-and-Transfer
  4. Dual Hallucination Learning Network
  5. Facial Pose Transfer Network
  6. Supervision-by-Hallucination-and-Transfer
  7. Experiments
  8. Datasets and Implementation Details
  9. Comparisons with SOTA Methods on CelebA and Helen Datasets
  10. Comparisons with SOTA Methods on the other Datasets
  11. Ablation Studies
  12. Self Evaluations
  13. Experimental Results and Discussions
  14. Conclusion

Figures (7)

  • Figure 1: Comparison of facial landmark detection results for images with different resolutions. The facial structure (e.g. face contour) of low-resolution face images (e.g. 16x16) is destroyed, which results in imprecise landmark detection (i.e., NME=23.19). While by taking low-resolution face images as inputs, our proposed SHT can learn high-resolution representations, which helps recover/preserve good facial structure and local details for precise low-resolution facial landmark detection.
  • Figure 2: The overall architecture of the proposed SHT. SHT contains two modules: Dual Hallucination Learning Network (DHLN) and Facial Pose Transfer Network (FPTN). By seamlessly integrating the DHLN and FPTN in a novel weakly-supervised framework, our SHT can simultaneously achieve more effective facial landmark detection and face hallucination.
  • Figure 3: The overall architecture of the proposed DHLN. By deep collaboration of landmark heatmap hallucination stream and face hallucination stream, DHLN can learn high-resolution representations with low-resolution input to recover/preserve good facial structure and local details, which help achieve more accurate FLD. Meanwhile, facial structure information learned from the landmark heatmap hallucination stream can also be applied to boost the face hallucination stream.
  • Figure 4: Gradient maps. The gradient maps ( corresponding to 3 (RGB) channels) contain rich facial structure information which helps hallucinate more realistic face images.
  • Figure 5: The main structure of FPTN. By transforming a person's face from one pose to another, FPTN can improve landmark heatmaps and faces hallucinated by DHLN for achieving more effective FLD and face hallucination.
  • ...and 2 more figures