CausalAffect: Causal Discovery for Facial Affective Understanding

TL;DR

CausalAffect addresses the challenge of learning psychologically plausible causal structures among facial Action Units (AUs) and their expressions without requiring jointly annotated data. It introduces a unified framework that combines a global, polarity-aware directed graph with per-sample refinements via a heterogeneous attention mechanism, and enforces true causality through feature-level counterfactual interventions in the AU latent space. The approach yields state-of-the-art results on six benchmarks for AU detection and expression recognition, while revealing both canonical and novel inhibitory relationships consistent with psychological theory. The work advances interpretable, weakly supervised causal discovery in facial affect analysis and provides insights for applications in HCI, robotics, and affective computing research.

Abstract

Understanding human affect from facial behavior requires not only accurate recognition but also structured reasoning over the latent dependencies that drive muscle activations and their expressive outcomes. Although Action Units (AUs) have long served as the foundation of affective computing, existing approaches rarely address how to infer psychologically plausible causal relations between AUs and expressions directly from data. We propose CausalAffect, the first framework for causal graph discovery in facial affect analysis. CausalAffect models AU-AU and AU-Expression dependencies through a two-level polarity and direction aware causal hierarchy that integrates population-level regularities with sample-adaptive structures. A feature-level counterfactual intervention mechanism further enforces true causal effects while suppressing spurious correlations. Crucially, our approach requires neither jointly annotated datasets nor handcrafted causal priors, yet it recovers causal structures consistent with established psychological theories while revealing novel inhibitory and previously uncharacterized dependencies. Extensive experiments across six benchmarks demonstrate that CausalAffect advances the state of the art in both AU detection and expression recognition, establishing a principled connection between causal discovery and interpretable facial behavior. All trained models and source code will be released upon acceptance.

Figures (3)

• Figure 1: Overview of CausalAffect, consisting of four key components: (1) Disentangled AU Bottleneck Encoding, which extracts semantically independent AU features; (2) Global Causal Graph Construction, which learns a polarity-aware acyclic graph capturing population-level dependencies; (3) Sample-Adaptive Graph Refinement, which adjusts the global graph for each instance via residual adaptation; and (4) Feature-Level Counterfactual Intervention, which enforces causal inductive bias through latent perturbations and factual–counterfactual consistency. The figure illustrates the AU$\rightarrow$AU branch; the AU$\rightarrow$Expression branch is analogous and omitted for brevity.
• Figure 2: Comparison of CausalAffect-learned causal relations (trained on All-DB) with existing approaches. For AU$\rightarrow$Expression relations, we compare against FACS ekman1978facial, cognitive priors du2014compound, statistical co-occurrence from Aff-Wild2 kollias2024distribution, and GNN-learned correlations kipf2016semi. For AU$\rightarrow$AU relations, we compare co-occurrence statistics derived from four commonly used datasets.
• Figure 3: Sample-adaptive causal relations inferred for individual cases. Examples are shown for AU$\rightarrow$Expression (top, blue) and AU$\rightarrow$AU (bottom, yellow), illustrating how CausalAffect adapts to context-specific structures. For clarity, only subgraphs involving the predicted expression or active AUs are visualized.