MGRR-Net: Multi-level Graph Relational Reasoning Network for Facial Action Units Detection

TL;DR

MGRR-Net tackles facial AU detection by jointly modeling dynamic interactions among regional AU patches and multi-level global face features. It introduces a region-level dynamic AU graph with a learnable adjacency, and two MH-GATs to extract channel- and pixel-level global attention, followed by hierarchical fusion and iterative refinement. The approach is trained end-to-end with AU detection and face-alignment objectives, achieving state-of-the-art results on DISFA and BP4D without external pretraining. This framework enhances robustness to expression and individual variability and offers a scalable method for precise AU discrimination with practical implications for clinical and behavioral analysis.

Abstract

The Facial Action Coding System (FACS) encodes the action units (AUs) in facial images, which has attracted extensive research attention due to its wide use in facial expression analysis. Many methods that perform well on automatic facial action unit (AU) detection primarily focus on modeling various types of AU relations between corresponding local muscle areas, or simply mining global attention-aware facial features, however, neglect the dynamic interactions among local-global features. We argue that encoding AU features just from one perspective may not capture the rich contextual information between regional and global face features, as well as the detailed variability across AUs, because of the diversity in expression and individual characteristics. In this paper, we propose a novel Multi-level Graph Relational Reasoning Network (termed MGRR-Net) for facial AU detection. Each layer of MGRR-Net performs a multi-level (i.e., region-level, pixel-wise and channel-wise level) feature learning. While the region-level feature learning from local face patches features via graph neural network can encode the correlation across different AUs, the pixel-wise and channel-wise feature learning via graph attention network can enhance the discrimination ability of AU features from global face features. The fused features from the three levels lead to improved AU discriminative ability. Extensive experiments on DISFA and BP4D AU datasets show that the proposed approach achieves superior performance than the state-of-the-art methods.

Figures (5)

• Figure 1: Comparisons between the proposed method and two state-of-the-art methods in AU feature learning, and the corresponding visualized activation maps for AU10 (Upper Lip Raiser / Levator labii superioris). (a) ARL shao2019facial performs global feature learning, (b) J$\rm \hat{A}$ANet shao2021jaa learns from predefined local regions based on the landmarks, and (c) multi-level feature learning from both local regions and global face regions (best viewed in color).
• Figure 2: The overall architecture of the proposed MGRR-Net for facial AU detection. Given one face image, the region-level features of local AU patches are extracted based on the detected landmarks from an efficient landmark localization network. The original global feature is extracted from the same shared stem network. Then the region-level GNN initialized with prior knowledge is applied to encode the correlation between different AU patches. Two separate MH-GATs are adopted to get two levels of global attention-aware features to supplement each AU. Finally, multiple levels of local-global features are fused by a hierarchical gated fusion strategy and refined by multiple iterations (best viewed in color).
• Figure 3: Box Plots of the distribution of performances on all AU categories (the labeled values are medians). (a) on DISFA 3-flod test set and (b) on BP4D 3-flod test set.
• Figure 4: Class activation maps that show the discriminative regions for different AUs in terms of different expressions and individuals on DISFA and BP4D datasets. We show the region center positions defined by the detected landmarks for the corresponding AUs. Abnormally shifted AU activation maps are marked with red boxes.
• Figure 5: Visualizations of the predefined AU correlation (a) and the learned relevance matrix (b) for the individual on BP4D. The corresponding class activation maps are shown in the first row of Figure \ref{['fig:visual_01']}.