FGTBT: Frequency-Guided Task-Balancing Transformer for Unified Facial Landmark Detection

TL;DR

The paper addresses robust facial landmark detection when training across multiple datasets with differing annotations and under challenging conditions such as large pose and illumination changes. It presents FGTBT, a Transformer-based framework that integrates a Fine-Grained Multi-Task Balancing loss (FMB-loss) and a Frequency-Guided Structure-Aware (FGSA) module to improve cross-dataset supervision and structural perception. The FMB-loss assigns per-landmark weights based on cross-dataset occurrence, while FGSA injects and regularizes high-frequency structural cues, yielding competitive state-of-the-art results on AFLW, 300W, COFW, and WFLW. While the approach increases computational cost, it offers a unified modeling paradigm that avoids dataset-specific models and enhances robustness for real-world face analysis tasks.

Abstract

Recently, deep learning based facial landmark detection (FLD) methods have achieved considerable success. However, in challenging scenarios such as large pose variations, illumination changes, and facial expression variations, they still struggle to accurately capture the geometric structure of the face, resulting in performance degradation. Moreover, the limited size and diversity of existing FLD datasets hinder robust model training, leading to reduced detection accuracy. To address these challenges, we propose a Frequency-Guided Task-Balancing Transformer (FGTBT), which enhances facial structure perception through frequency-domain modeling and multi-dataset unified training. Specifically, we propose a novel Fine-Grained Multi-Task Balancing loss (FMB-loss), which moves beyond coarse task-level balancing by assigning weights to individual landmarks based on their occurrence across datasets. This enables more effective unified training and mitigates the issue of inconsistent gradient magnitudes. Additionally, a Frequency-Guided Structure-Aware (FGSA) model is designed to utilize frequency-guided structure injection and regularization to help learn facial structure constraints. Extensive experimental results on popular benchmark datasets demonstrate that the integration of the proposed FMB-loss and FGSA model into our FGTBT framework achieves performance comparable to state-of-the-art methods. The code is available at https://github.com/Xi0ngxinyu/FGTBT.

Figures (7)

• Figure 1: The proposed FGTBT enables unified training across multiple facial landmark detection datasets with different annotation protocols and achieves superior performance.
• Figure 2: The overall architecture of the proposed FGTBT. The input image is first processed through a Patch Embedding layer, followed by four hierarchical Transformer stages connected via downsampling. At each stage, frequency-domain information is first utilized to generate structure features, which are then effectively integrated into the original model using the structure injection and regularization. The features at each stage are finally decoded into facial landmarks with a decoder head. By explicitly incorporating frequency-domain structure guidance, FGTBT significantly improves structural awareness and face alignment accuracy.
• Figure 3: The landmark annotation distributions across four datasets: AFLW, WFLW, 300W, and COFW. Red dots represent landmarks that appear in all four datasets, while orange, blue, and green denote landmarks shared by three, two, and one dataset(s), respectively. This visualization highlights a challenge in multi-dataset training: landmarks that appear in more datasets tend to receive more attention during learning, while those appearing less frequently are more prone to neglect.
• Figure 4: The high-frequency information extraction. Given an input image, it is first transformed into the frequency domain via the Fast Fourier Transform (FFT). A high-frequency emphasis mask $M_h$ is then applied element-wise to the spectrum, followed by an inverse FFT to reconstruct a spatial-domain image $I_{hf}$ that predominantly contains high-frequency components. For example, the resulting image clearly highlights the structural information of faces with large poses, thereby providing valuable cues for enhancing FLD performance.
• Figure 5: The average loss of landmarks that appear with different frequencies across the four datasets under different values of $\beta$. (a) presents the results on the 300W-chall testset, while (b) shows the results on the WFLW testset.