Facing Asymmetry -- Uncovering the Causal Link between Facial Symmetry and Expression Classifiers using Synthetic Interventions

TL;DR

This work uses insights from causal reasoning to investigate the hypothesis that one crucial factor guiding the internal decision rules is facial symmetry, and develops a synthetic interventional framework to analyze how facial symmetry impacts a network's output behavior while keeping other factors fixed.

Abstract

Understanding expressions is vital for deciphering human behavior, and nowadays, end-to-end trained black box models achieve high performance. Due to the black-box nature of these models, it is unclear how they behave when applied out-of-distribution. Specifically, these models show decreased performance for unilateral facial palsy patients. We hypothesize that one crucial factor guiding the internal decision rules is facial symmetry. In this work, we use insights from causal reasoning to investigate the hypothesis. After deriving a structural causal model, we develop a synthetic interventional framework. This approach allows us to analyze how facial symmetry impacts a network's output behavior while keeping other factors fixed. All 17 investigated expression classifiers significantly lower their output activations for reduced symmetry. This result is congruent with observed behavior on real-world data from healthy subjects and facial palsy patients. As such, our investigation serves as a case study for identifying causal factors that influence the behavior of black-box models.

Figures (26)

• Figure 1: Expression classifier structural causal model: $Y$ is the expression influenced by the latent distribution of all facial images (hatched box), $S_*$ samples from this latent distribution reimers2020determining. $\mathcal{D}_{\text{train}}$ is the training data distribution. The model architecture is an exogenous variable, and weights $\theta$ are learned using an optimizer, i.e., a training algorithm. The model's predictions $\hat{Y}$ result from the trained model $\mathbb{F}_\theta$. We investigate whether $\mathbb{F}_\theta$ is independent of the model predictions $\hat{Y}$ ( dashed red arrow). Additionally, we analyze the changes in behavior for varying facial symmetries. Toward this goal, we perform synthetic interventions ($do(\text{Facial Symmetry}:= s)$) on facial symmetry variable using 3d morphable models $I_{\varphi^{(e)}}$. Note that these $I_{\varphi^{(e)}}$ are a part (subpopulation) of the latent distribution of all facial images. Adapted from Figure 2 in reimers2020determining.
• Figure 2: We display the optimized synthetic face images $I_{\varphi^{(e)}}$ for the neutral expression (\ref{['fig:rmn_neutral']}) and for the six base emotions (\ref{['fig:rmn_angry']}) - (\ref{['fig:rmn_surprise']}) based on the ResidualMaskingNet classifier pham2021facial. Furthermore, we simulate with our geometric face model $\mathcal{G}_{s,t}(\cdot)$ different interpolations $t$ for a symmetry of $s=0.0$. At $t=0.0$ (\ref{['fig:hse_happy_s0_t00']}) we have a neutral expression morphing into an asymmetric happy expression at $t=1.0$ (\ref{['fig:hse_happy_s0_t89']}).
• Figure 3: Visualization of our impact score for a classifier's happy logit activation: In a synthetic setting, a model was shown a face transition from neutral to a happy expression (\ref{['fig:fais_vis_a']}). A model would be invariant toward changes along the symmetry axis if $\nabla_{s}\mathbb{F} = 0$. However, the actual activation logits (happy) show a lower activation (\ref{['fig:fais_vis_b']}). This is more evident in the visualization of the estimated $\nabla_s$ in (\ref{['fig:fais_vis_c']}).
• Figure 4: We display two example faces per analyzed group (synthetic, probands, and patients). Both healthy probands and patients with unilateral facial palsy were instructed to mimic the shown emotions, similar to the FER2013 benchmark dumitru2013fer.
• Figure 5: We display a model's activation (mean and std.) curve at $t=1.0$ for each expression recognition dataset. Note that the x-axis is inverted, so we start with high symmetry. Lower symmetry generally results in lower logit activations across all expressions, with hatched lines indicating misclassification. We show the surface variants, such as \ref{['fig:fais_vis_a']}, in the supplementary material.

Theorems & Definitions (1)

• definition thmcounterdefinition: Structural Causal Model in pearl2009causality and bareinboim2022onpearl