Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Micro-AU CLIP: Fine-Grained Contrastive Learning from Local Independence to Global Dependency for Micro-Expression Action Unit Detection

Jinsheng Wei, Fengzhou Guo, Yante Li, Haoyu Chen, Guanming Lu, Guoying Zhao

Abstract

Micro-expression (ME) action units (Micro-AUs) provide objective clues for fine-grained genuine emotion analysis. Most existing Micro-AU detection methods learn AU features from the whole facial image/video, which conflicts with the inherent locality of AU, resulting in insufficient perception of AU regions. In fact, each AU independently corresponds to specific localized facial muscle movements (local independence), while there is an inherent dependency between some AUs under specific emotional states (global dependency). Thus, this paper explores the effectiveness of the independence-to-dependency pattern and proposes a novel micro-AU detection framework, micro-AU CLIP, that uniquely decomposes the AU detection process into local semantic independence modeling (LSI) and global semantic dependency (GSD) modeling. In LSI, Patch Token Attention (PTA) is designed, mapping several local features within the AU region to the same feature space; In GSD, Global Dependency Attention (GDA) and Global Dependency Loss (GDLoss) are presented to model the global dependency relationships between different AUs, thereby enhancing each AU feature. Furthermore, considering CLIP's native limitations in micro-semantic alignment, a microAU contrastive loss (MiAUCL) is designed to learn AU features by a fine-grained alignment of visual and text features. Also, Micro-AU CLIP is effectively applied to ME recognition in an emotion-label-free way. The experimental results demonstrate that Micro-AU CLIP can fully learn fine-grained micro-AU features, achieving state-of-the-art performance.

Micro-AU CLIP: Fine-Grained Contrastive Learning from Local Independence to Global Dependency for Micro-Expression Action Unit Detection

Abstract

Micro-expression (ME) action units (Micro-AUs) provide objective clues for fine-grained genuine emotion analysis. Most existing Micro-AU detection methods learn AU features from the whole facial image/video, which conflicts with the inherent locality of AU, resulting in insufficient perception of AU regions. In fact, each AU independently corresponds to specific localized facial muscle movements (local independence), while there is an inherent dependency between some AUs under specific emotional states (global dependency). Thus, this paper explores the effectiveness of the independence-to-dependency pattern and proposes a novel micro-AU detection framework, micro-AU CLIP, that uniquely decomposes the AU detection process into local semantic independence modeling (LSI) and global semantic dependency (GSD) modeling. In LSI, Patch Token Attention (PTA) is designed, mapping several local features within the AU region to the same feature space; In GSD, Global Dependency Attention (GDA) and Global Dependency Loss (GDLoss) are presented to model the global dependency relationships between different AUs, thereby enhancing each AU feature. Furthermore, considering CLIP's native limitations in micro-semantic alignment, a microAU contrastive loss (MiAUCL) is designed to learn AU features by a fine-grained alignment of visual and text features. Also, Micro-AU CLIP is effectively applied to ME recognition in an emotion-label-free way. The experimental results demonstrate that Micro-AU CLIP can fully learn fine-grained micro-AU features, achieving state-of-the-art performance.
Paper Structure (29 sections, 13 equations, 7 figures, 10 tables)

This paper contains 29 sections, 13 equations, 7 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Micro-AU Detection Paradigm
  4. Contrastive Learning
  5. MER Learning Way
  6. Proposed Method
  7. Visual and Text Backbone
  8. LSI Module
  9. Fine-Grained Local Feature Partitioning
  10. Patch Token Attention
  11. GSD Module
  12. Global Dependency Attention
  13. Global Dependency Attention Loss
  14. MiAUC Loss
  15. Emotion-Label-Free MER
  16. ...and 14 more sections

Figures (7)

  • Figure 1: The illustration of the local semantic independence and global semantic dependence.
  • Figure 2: Framework of the proposed Micro-AU CLIP. (a) ViT-based visual backbone extracts patch-level visual tokens from the input image.; (b) Text backbone encodes AU-related textual prompts into text features; (c) In LSI, the model utilizes PTA to learn each AU feature individually; (d) GSD models the global dependence between different AUs and propagates the global context information to the features of each AU.
  • Figure 3: Illustration of AU and the corresponding facial landmarks.
  • Figure 4: The details of PTA and GDA.
  • Figure 5: Attention heatmap visualization results
  • ...and 2 more figures