Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Proto-Former: Unified Facial Landmark Detection by Prototype Transformer

Shengkai Hu, Haozhe Qi, Jun Wan, Jiaxing Huang, Lefei Zhang, Hang Sun, Dacheng Tao

TL;DR

Proto-Former tackles cross-dataset facial landmark detection by unifying annotations into 124 shared landmarks and learning dataset-specific prototypes through an Adaptive Prototype-Aware Encoder (APAE). A Progressive Prototype-Aware Decoder (PPAD) then uses prototype-guided prompts to refine landmark queries, with a Prototype-Aware loss (L_{PA}) mitigating gradient conflicts during multi-dataset training. The approach achieves state-of-the-art results on 300W, COFW, WFLW, and AFLW, while handling occlusions, large pose variations, and blur via multi-scale prototypes and prompt-guided decoding. Overall, Proto-Former enables a single model to predict varying landmark sets accurately across datasets, improving efficiency and generalization for practical FLD deployment.

Abstract

Recent advances in deep learning have significantly improved facial landmark detection. However, existing facial landmark detection datasets often define different numbers of landmarks, and most mainstream methods can only be trained on a single dataset. This limits the model generalization to different datasets and hinders the development of a unified model. To address this issue, we propose Proto-Former, a unified, adaptive, end-to-end facial landmark detection framework that explicitly enhances dataset-specific facial structural representations (i.e., prototype). Proto-Former overcomes the limitations of single-dataset training by enabling joint training across multiple datasets within a unified architecture. Specifically, Proto-Former comprises two key components: an Adaptive Prototype-Aware Encoder (APAE) that performs adaptive feature extraction and learns prototype representations, and a Progressive Prototype-Aware Decoder (PPAD) that refines these prototypes to generate prompts that guide the model's attention to key facial regions. Furthermore, we introduce a novel Prototype-Aware (PA) loss, which achieves optimal path finding by constraining the selection weights of prototype experts. This loss function effectively resolves the problem of prototype expert addressing instability during multi-dataset training, alleviates gradient conflicts, and enables the extraction of more accurate facial structure features. Extensive experiments on widely used benchmark datasets demonstrate that our Proto-Former achieves superior performance compared to existing state-of-the-art methods. The code is publicly available at: https://github.com/Husk021118/Proto-Former.

Proto-Former: Unified Facial Landmark Detection by Prototype Transformer

TL;DR

Proto-Former tackles cross-dataset facial landmark detection by unifying annotations into 124 shared landmarks and learning dataset-specific prototypes through an Adaptive Prototype-Aware Encoder (APAE). A Progressive Prototype-Aware Decoder (PPAD) then uses prototype-guided prompts to refine landmark queries, with a Prototype-Aware loss (L_{PA}) mitigating gradient conflicts during multi-dataset training. The approach achieves state-of-the-art results on 300W, COFW, WFLW, and AFLW, while handling occlusions, large pose variations, and blur via multi-scale prototypes and prompt-guided decoding. Overall, Proto-Former enables a single model to predict varying landmark sets accurately across datasets, improving efficiency and generalization for practical FLD deployment.

Abstract

Recent advances in deep learning have significantly improved facial landmark detection. However, existing facial landmark detection datasets often define different numbers of landmarks, and most mainstream methods can only be trained on a single dataset. This limits the model generalization to different datasets and hinders the development of a unified model. To address this issue, we propose Proto-Former, a unified, adaptive, end-to-end facial landmark detection framework that explicitly enhances dataset-specific facial structural representations (i.e., prototype). Proto-Former overcomes the limitations of single-dataset training by enabling joint training across multiple datasets within a unified architecture. Specifically, Proto-Former comprises two key components: an Adaptive Prototype-Aware Encoder (APAE) that performs adaptive feature extraction and learns prototype representations, and a Progressive Prototype-Aware Decoder (PPAD) that refines these prototypes to generate prompts that guide the model's attention to key facial regions. Furthermore, we introduce a novel Prototype-Aware (PA) loss, which achieves optimal path finding by constraining the selection weights of prototype experts. This loss function effectively resolves the problem of prototype expert addressing instability during multi-dataset training, alleviates gradient conflicts, and enables the extraction of more accurate facial structure features. Extensive experiments on widely used benchmark datasets demonstrate that our Proto-Former achieves superior performance compared to existing state-of-the-art methods. The code is publicly available at: https://github.com/Husk021118/Proto-Former.
Paper Structure (33 sections, 15 equations, 7 figures, 9 tables)

This paper contains 33 sections, 15 equations, 7 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Related Work
  4. Method
  5. Unified Facial Landmark Detection (UFLD)
  6. Unified Landmark Index
  7. Unified Landmark Matching
  8. Overall Architecture
  9. Adaptive prototype-Aware Encoder (APAE)
  10. Adaptive Prototype Extractor
  11. Prototype Encoder
  12. Progressive Prototype-Aware Decoder (PPAD)
  13. Prompt Generator
  14. Fusion Block
  15. Fusion Block
  16. ...and 18 more sections

Figures (7)

  • Figure 1: (a) Independent Model (IM), where IM.1, IM.2 and IM.K are independent. (b) Model with independent Encoders (ME), whose encoders (IM-E.1, IM-E.2 and IM-E.K) are independent. (c) Unified Model (UM), where UM-E and UM-D are the unified modules. (d) Our proposed unified Proto-Former is able to extract dataset-specific features similar to (a) by collaborating multiple adaptive discriminative encoders and an additional prompt block in the decoder. (e) Comparison with state-of-the-art FLD methods on four popular datasets 300W, COFW, WFLW, and AFLW.
  • Figure 2: The network structure of the proposed Proto-Former. The image is processed through the backbone and APAE to handle features from different datasets. Reshaped features are fed into to the PPAD for dimensional information extraction, which enables High-resolution prototypes and landmark queries to be combined in the Prompt Generator to produce and inject prompts into the Proto-Decoder. Finally, the landmark query is used to predict the unified landmark index and its corresponding coordinates via the FC layer and the MLP layer.
  • Figure 3: (a) The original landmark index. (b) the proposed unified landmark index. To achieve unified facial landmark detection, we combined the original landmarks from four popular datasets into 124 unified landmarks.
  • Figure 4: The proposed prompt generator and fusion block primarily consist of a prompt generation mechanism based on the similarity matrix and a fusion process that integrates the original landmark queries with the generated prompts. This design effectively enhances the model’s attention to facial structural features, leading to improved performance.
  • Figure 5: Comparison of prototypes and feature maps in normal (the first 4 rows) and complex circumstances (the last 3 rows). (a) predicted landmarks, (b) low-resolution feature map, (c) low-resolution prototype, (d) high-resolution feature map and (e) high-resolution prototype. It demonstrates our proposed Proto-Former can extracts effective prototypes.
  • ...and 2 more figures