PriorProbe: Recovering Individual-Level Priors for Personalizing Neural Networks in Facial Expression Recognition

TL;DR

PriorProbe addresses the challenge of aligning neural network perception with individual human judgments by recovering both priors and likelihoods through a Bayesian, MCMC-based elicitation. It introduces DistFace to disentangle affect from identity, enabling efficient, low‑dimensional sampling of individual cognitive priors over emotion categories. By integrating the recovered priors with a state-of-the-art facial expression model, the approach yields substantial gains on ambiguous stimuli without degrading performance on ground-truth labels, demonstrating a general and interpretable path to personalization. This framework has broad implications for personalized AI systems and could enable targeted data generation and more stable user-specific adaptations in real-world settings.

Abstract

Incorporating individual-level cognitive priors offers an important route to personalizing neural networks, yet accurately eliciting such priors remains challenging: existing methods either fail to uniquely identify them or introduce systematic biases. Here, we introduce PriorProbe, a novel elicitation approach grounded in Markov Chain Monte Carlo with People that recovers fine-grained, individual-specific priors. Focusing on a facial expression recognition task, we apply PriorProbe to individual participants and test whether integrating the recovered priors with a state-of-the-art neural network improves its ability to predict an individual's classification on ambiguous stimuli. The PriorProbe-derived priors yield substantial performance gains, outperforming both the neural network alone and alternative sources of priors, while preserving the network's inference on ground-truth labels. Together, these results demonstrate that PriorProbe provides a general and interpretable framework for personalizing deep neural networks.

Figures (9)

• Figure 1: The framework of using PriorProbe to inform neural network with individual-level preferences. The left panel illustrates a typical PriorProbe procedure, and the right panel describe how the recovered individual prior informs a shared neural network about individual differences in inference.
• Figure 2: Generative performance of our model on the affect‑transfer task. The right column shows randomly sampled identities from the in‑distribution dataset (BU‑4DFE), and the top row shows target facial affects selected from an out‑of‑distribution dataset (CelebA). The remaining grid displays images reconstructed by our model, combining the identity from the corresponding right‑column exemplar with the facial affect from the corresponding top‑row exemplar. The performance indicates that the model has learned a facial affect representation space that is effectively disentangled from facial identity.
• Figure 3: Example individual priors recovered by PriorProbe. To preserve visual clarity and distinguishability across participants, we display priors from twelve randomly selected individuals. The black heptagon represents the average prior across individuals.
• Figure 4: Predictive performance on individual‑level categorization across models. ResEmoteNet is the neural network with SOTA performance in face expression recognition. The individual priors recovered by PriorProbe show the best performance in helping ResEmoteNet capture individual‑level responses to ambiguous stimuli.
• Figure 5: Predictive performance of ResEmoteNet (red line) and PriorProbe-informed ResEmoteNet (grey bars) on individual‑level classification of ambiguous faces across all participants.