Leveraging Synthetic Data for Generalizable and Fair Facial Action Unit Detection

TL;DR

The paper tackles limited labeled data and gender bias in facial AU detection by integrating synthetic data generation through expression retargeting with a multi-source domain adaptation framework called Paired Moment Matching (PM2). PM2 combines Maximum Classifier Discrepancy (MCD) for cross-domain alignment with paired real–synthetic moment matching to preserve AU discriminability while enforcing gender-aware fairness. Empirical results show PM2 improves cross-domain AU detection performance and fairness across BP4D→DISFA and BP4D→GFT, outperforming single-source baselines and prior MSDA approaches. The work demonstrates the potential of synthetic data to enhance real-world AU detection systems and highlights practical considerations for fairness in deployment, with plans to broaden subject attributes and automate data generation.

Abstract

Facial action unit (AU) detection is a fundamental block for objective facial expression analysis. Supervised learning approaches require a large amount of manual labeling which is costly. The limited labeled data are also not diverse in terms of gender which can affect model fairness. In this paper, we propose to use synthetically generated data and multi-source domain adaptation (MSDA) to address the problems of the scarcity of labeled data and the diversity of subjects. Specifically, we propose to generate a diverse dataset through synthetic facial expression re-targeting by transferring the expressions from real faces to synthetic avatars. Then, we use MSDA to transfer the AU detection knowledge from a real dataset and the synthetic dataset to a target dataset. Instead of aligning the overall distributions of different domains, we propose Paired Moment Matching (PM2) to align the features of the paired real and synthetic data with the same facial expression. To further improve gender fairness, PM2 matches the features of the real data with a female and a male synthetic image. Our results indicate that synthetic data and the proposed model improve both AU detection performance and fairness across genders, demonstrating its potential to solve AU detection in-the-wild.

Figures (7)

• Figure 1: Overview of the proposed Paired Moment Matching (PM2). In this work, the encoder is a Swin Transformer and the classifier is an MLP. PM2 has two modules: (1) Maximum Classifier Discrepancy (MCD) aligns the source and target by minimizing the discrepancy between two classifiers on target samples. (2) PM2 aligns the real and synthetic domains by minimizing the moment distances between the real-syn pairs. Each pair contains one real image and two synthetic images generated with a male and a female avatar, sharing the same facial expression.
• Figure 2: (Best viewed in color) Comparison of previous and proposed methods for domain alignment. Middle: Previous methods align the overall distributions and corrupt the class boundary. Right: Our method aligns the paired data with the same facial expression and maintains the class boundary.
• Figure 3: (Best viewed in color) Comparison of previous and proposed methods for bias mitigation. Left: The real domain has gender bias thus female and male distributions are not aligned. While the synthetic (syn.) data is balanced and the gender distributions are matched. Middle: Though previous methods match the real and synthetic domains well, they fail to align the gender distributions. Right: Our method aligns each real image with a male and a female avatar. The method explicitly aligns the real and synthetic, female and male domains.
• Figure 4: Examples of synthetically generated data through expression re-targeting. Real images come from BP4D. For each real subject, we generate the synthetic data with two avatars, one male and one female. The synthetic data has similar facial expressions as the real images.
• Figure 5: Occurrence rates of AU labels for different datasets. BP4D has the highest occurrence rate on average while DISFA and GFT are severely imbalanced.