Causal Intervention for Subject-Deconfounded Facial Action Unit Recognition

TL;DR

Subject variation biases facial AU recognition; the authors propose a structural causal model with four variables $(X,S,R,Y)$ to analyze the causal paths and introduce a plug-in CIS module that implements back-door adjustment to estimate $P(Y|do(X))$ via a confounder dictionary of subject prototypes. CISNet inserts CIS after the backbone to deconfound Subject, combining $f_{cur}$ and $r_{cur}$ through a linear mapping to predict AU probabilities, and uses an adaptive loss to handle AU class imbalance. The key contributions are the formal AU causal diagram, the back-door intervention formulation, and the CISNet architecture that can be plugged into existing models. Experiments on BP4D and DISFA demonstrate state-of-the-art F1-scores, validating that causal intervention reduces subject-related prediction bias and improves generalization across unseen subjects.

Abstract

Subject-invariant facial action unit (AU) recognition remains challenging for the reason that the data distribution varies among subjects. In this paper, we propose a causal inference framework for subject-invariant facial action unit recognition. To illustrate the causal effect existing in AU recognition task, we formulate the causalities among facial images, subjects, latent AU semantic relations, and estimated AU occurrence probabilities via a structural causal model. By constructing such a causal diagram, we clarify the causal effect among variables and propose a plug-in causal intervention module, CIS, to deconfound the confounder \emph{Subject} in the causal diagram. Extensive experiments conducted on two commonly used AU benchmark datasets, BP4D and DISFA, show the effectiveness of our CIS, and the model with CIS inserted, CISNet, has achieved state-of-the-art performance.

Figures (7)

• Figure 1: Illustration of subject variation problem. The AU semantic relations embedded in the facial images of the four subjects vary due to the differences among their customs of expressing happiness. Using samples collected from the first three subjects for training may cause the learned latent AU semantic relations containing subject-specific ones (dark yellow lines) in addition to universal ones (dark blue lines). When encountering a new subject (Subject 4) in the inference stage, differences between Subject 4's specific AU relations and the learned ones will lead to prediction bias.
• Figure 2: Illustration of our AU causal diagram.
• Figure 3: Overview. First, a facial image is fed into a backbone network for feature extraction. Instead of directly using the extracted feature $f_{\rm cur}$ for classification, we put $f_{\rm cur}$ into the proposed CIS module for causal intervention on Subject, i.e. approximation of $P(Y|do(X))$. In CIS module, the output of Approx. R---$r_{\rm cur}$ and $f_{\rm cur}$ are further fed into a linear layer separately and concatenated as the input of a classifier for AU prediction. The key component in CIS module is the approximation of $R$, which involves three parts for calculation, confounder attentions, a confounder dictionary, and confounder priors.
• Figure 4: Impact of the number of training subject.
• Figure 5: PCC among AUs for different subjects. From left to right, PCC matrices are computed based on the ground-truth AU labels, predicted ones using CISNet (w/ CIS), and predicted ones using the baseline model (w/o CIS), respectively. Numbers under PCC heatmaps are cosine similarities between themselves and the corresponding ground-truth.