Cascaded Dual Vision Transformer for Accurate Facial Landmark Detection

TL;DR

This paper addresses accurate facial landmark detection by modeling both spatial context and geometric relationships among landmarks. It introduces Dual Vision Transformer (D-ViT) that fuses spatial-split and channel-split ViTs and employs Long Skip Connections to support deeper cascaded prediction blocks. The approach treats the channel dimension as heatmap bases and uses self-attention to learn their interdependencies, enabling robust predictions under occlusion and pose variation. Experiments on WFLW, COFW, and 300W demonstrate state-of-the-art performance and strong generalization, with ablations validating the design choices.

Abstract

Facial landmark detection is a fundamental problem in computer vision for many downstream applications. This paper introduces a new facial landmark detector based on vision transformers, which consists of two unique designs: Dual Vision Transformer (D-ViT) and Long Skip Connections (LSC). Based on the observation that the channel dimension of feature maps essentially represents the linear bases of the heatmap space, we propose learning the interconnections between these linear bases to model the inherent geometric relations among landmarks via Channel-split ViT. We integrate such channel-split ViT into the standard vision transformer (i.e., spatial-split ViT), forming our Dual Vision Transformer to constitute the prediction blocks. We also suggest using long skip connections to deliver low-level image features to all prediction blocks, thereby preventing useful information from being discarded by intermediate supervision. Extensive experiments are conducted to evaluate the performance of our proposal on the widely used benchmarks, i.e., WFLW, COFW, and 300W, demonstrating that our model outperforms the previous SOTAs across all three benchmarks.

Figures (6)

• Figure 1: Architecture of our framework. Long skip connections (depicted as upper black lines) distribute low-level features to each prediction block, thereby preventing useful information from being discarded by intermediate supervision. D-ViT learns contextual features of the image through spatially split patches and captures the underlying geometric relations among landmarks using channel-split features.
• Figure 2: Heatmap is linear combination of channel-split features. $\bigotimes$ denotes dot-product. Intermediate supervision uses Conv2D with $1\times1$ kernel to convert the features extracted from a prediction block into heatmaps. This implies channel-split features linearly expand the heatmap space. Based on this observation, we take advantage of the self-attention mechanism to learn underlying geometric relations among landmarks via channel-split features.
• Figure 3: Implementation details of our D-ViT.
• Figure 4: Qualitative results of different prediction blocks on WFLW dataset. Green and red points represent the predicted and ground-truth landmarks, respectively. Orange or Yellow circles indicate the clear failures, which can be improved with help of D-ViT.
• Figure 5: Illustrations of alternative skip connections. ResCBSP denotes the residual connections between two sequential prediction blocks. DenC denotes dense connections where any two prediction blocks have a skip connection. Refer to \ref{['fig:model']} for definition of prediction block and colorful rectangles.