Modeling Cultural Bias in Facial Expression Recognition with Adaptive Agents

TL;DR

This work tackles the robustness of facial expression recognition (FER) under cross-cultural variation and perceptual degradation by introducing an agent-based streaming benchmark. Agents operate on a 5x5 lattice using frozen CLIP embeddings augmented by online adapters trained at initial, clean conditions, then evaluated under a sigma-scheduled Gaussian blur. By comparing monocultural and mixed populations (balanced and imbalanced), the study reveals asymmetric degradation patterns: JAFFE-based agents are more brittle under blur, while KDEF-based agents show more gradual decline; mixed populations exhibit intermediate behaviors shaped by contact structure and composition. The findings highlight the influence of cultural composition and interaction dynamics on FER robustness, with implications for fairness and human–machine interaction in diverse real-world settings and motivating domain-aware adaptation and balanced embedding strategies.

Abstract

Facial expression recognition (FER) must remain robust under both cultural variation and perceptually degraded visual conditions, yet most existing evaluations assume homogeneous data and high-quality imagery. We introduce an agent-based, streaming benchmark that reveals how cross-cultural composition and progressive blurring interact to shape face recognition robustness. Each agent operates in a frozen CLIP feature space with a lightweight residual adapter trained online at sigma=0 and fixed during testing. Agents move and interact on a 5x5 lattice, while the environment provides inputs with sigma-scheduled Gaussian blur. We examine monocultural populations (Western-only, Asian-only) and mixed environments with balanced (5/5) and imbalanced (8/2, 2/8) compositions, as well as different spatial contact structures. Results show clear asymmetric degradation curves between cultural groups: JAFFE (Asian) populations maintain higher performance at low blur but exhibit sharper drops at intermediate stages, whereas KDEF (Western) populations degrade more uniformly. Mixed populations exhibit intermediate patterns, with balanced mixtures mitigating early degradation, but imbalanced settings amplify majority-group weaknesses under high blur. These findings quantify how cultural composition and interaction structure influence the robustness of FER as perceptual conditions deteriorate.

Figures (7)

• Figure 1: Initial snapshot (t = 1) of the agent lattice. Agents occupy grid cells as colored circles: color indicates the current emotion (e.g., happy, sad, angry), and the text inside each circle denotes the agent’s ID and the elicited emotion.
• Figure 2: Agent lattice at t = 5. Circle size reflects confidence in their emotion predictions: repeated misclassifications shrink the circle, while each correct prediction expands it back toward its original size.
• Figure 3: Gaussian blur levels ($\sigma$). The same face shown with increasing blur from left to right: $\sigma = 0,1,2,3,4$. As $\sigma$ increases, edges soften and fine facial details disappear, making emotion classification more difficult.
• Figure 4: Macro-F1 evolution for monocultural populations (KDEF-only vs. JAFFE-only) during learning and evaluation with increasing blur $\sigma \in \{0,1,2,3,4\}$. Red vertical line marks the end of the learning phase.
• Figure 5: Macro-F1 evolution for a balanced mixed population (5 KDEF / 5 JAFFE). Curves show intra-cultural (same group), cross-cultural (different groups), and global averages as blur $\sigma$ increases.