Adaptively Enhancing Facial Expression Crucial Regions via Local Non-Local Joint Network

TL;DR

This work tackles facial expression recognition under small inter-class differences by eliminating the need for manually annotated facial landmarks. It introduces LNLAttenNet, a dual-branch architecture that combines a Non-Local Attention Network operating on global U-Net feature maps with a Local Multi-Networks Ensemble that processes dense local patches, with non-local weights guiding the local ensemble. The networks are jointly optimized, using a fusion of the non-local vector $\mathbf{g}^*$ and the local ensemble vector $\mathbf{f}_{en}$ through three fully connected layers, with loss $L=\mathcal{L}_{entropy}+\gamma L_2$ and $\gamma=0.0001$, achieving strong performance across five FER datasets. Experiments show the model automatically highlights discriminative regions (e.g., around the eyes and mouth) without landmark information, yielding competitive results and offering a new direction for landmark-free FER design.

Abstract

Facial expression recognition (FER) is still one challenging research due to the small inter-class discrepancy in the facial expression data. In view of the significance of facial crucial regions for FER, many existing researches utilize the prior information from some annotated crucial points to improve the performance of FER. However, it is complicated and time-consuming to manually annotate facial crucial points, especially for vast wild expression images. Based on this, a local non-local joint network is proposed to adaptively light up the facial crucial regions in feature learning of FER in this paper. In the proposed method, two parts are constructed based on facial local and non-local information respectively, where an ensemble of multiple local networks are proposed to extract local features corresponding to multiple facial local regions and a non-local attention network is addressed to explore the significance of each local region. Especially, the attention weights obtained by the non-local network is fed into the local part to achieve the interactive feedback between the facial global and local information. Interestingly, the non-local weights corresponding to local regions are gradually updated and higher weights are given to more crucial regions. Moreover, U-Net is employed to extract the integrated features of deep semantic information and low hierarchical detail information of expression images. Finally, experimental results illustrate that the proposed method achieves more competitive performance compared with several state-of-the art methods on five benchmark datasets. Noticeably, the analyses of the non-local weights corresponding to local regions demonstrate that the proposed method can automatically enhance some crucial regions in the process of feature learning without any facial landmark information.

Figures (14)

• Figure 1: An illustration of facial crucial regions from six expressions, where two facial images (ID1 and ID2) are shown for each expression. The regions around eyes and mouths are cropped as examples of FCRs in the purple box and the green box, respectively.
• Figure 2: Schematic diagram of the pixel deviations at image level when posture changing. To demonstrate this change, we measured the movement of 68 landmark points on the faces with different postures and the same identity. In figure (a) and (b), 68 landmark points are marked with a green cross, and figure (c) shows the movement of 68 landmark points.
• Figure 3: A simple view of the proposed model (LNLAttenNet). The part in the green dotted box shows the global weights corresponding to 16 local regions (from Patch 1 to Patch 16) obtained by LNLAttenNet, and the part under the green dotted box is a simple framework of LNLAttenNet.
• Figure 4: The framework of the proposed model (LNLAttenNet). LNLAttenNet uses U-Net to generate feature map with the same resolution as the input image. Then, its feature map (Conv9-2) is cropped into $M$ local patches to construct the local multi-networks ensemble model, where each patch is used to generate an individual network based on the structure of Simple Net. The feature map (Conv5-2) is used to construct the global attention network. Finally, the global and local features are integrated based on the global weights, and then three fully connected layers are followed.
• Figure 5: Overview of the Non-Local attention model.