Beyond One-Size-Fits-All: A Survey of Personalized Affective Computing in Human-Agent Interaction

TL;DR

Personalization in affective computing is essential to account for individual differences in emotional expression within Human-Agent Interaction. The authors provide the first comprehensive taxonomy, separating data-level and model-level approaches and detailing seven subcategories with problem formulations, complemented by a statistical analysis of 101 papers across tasks, interaction modes, and contexts. The survey highlights trends toward data-grouping methods and rising model-level techniques, alongside a gradual uptake of deep learning and generative approaches, while identifying gaps in fairness, dataset availability, and evaluation. Together, these insights offer a practical roadmap for researchers to advance personalized affective systems across HCI, HHI, and HRI contexts, including considerations of bias, datasets, and the potential role of generative AI.

Abstract

In personalized machine learning, the aim of personalization is to train a model that caters to a specific individual or group of individuals by optimizing one or more performance metrics and adhering to specific constraints. In this paper, we discuss the need for personalization in affective computing and present the first survey of existing approaches for personalization in affective computing. Our review spans training techniques and objectives towards the personalization of affective computing models across various interaction modes and contexts. We develop a taxonomy that clusters existing approaches into Data-level and Model-level approaches. Across the Data-Level and Model-Level broad categories, we group existing approaches into seven sub-categories: (1) User-Specific Models, (2) Group-Specific Models, (3) Weighting-Based Approaches, (4) Feature Augmentation, (5) Generative-Based Models which fall into the Data-Level approaches, (6) Fine-Tuning Approaches, and (7) Multitask Learning Approaches falling under the model-level approaches. We provide a problem formulation for personalized affective computing, and to each of the identified sub-categories. Additionally, we provide a statistical analysis of the surveyed literature, analyzing the prevalence of different affective computing tasks, interaction modes (i.e. Human-Computer Interaction (HCI), Human-Human interaction (HHI), Human-Robot Interaction (HRI)), interaction contexts (e.g. educative, social, gaming, etc.), and the level of personalization among the surveyed works. Based on our analysis, we provide a road-map for researchers interested in exploring this direction.

Figures (14)

• Figure 1: Example of some personal factors that may have an effect on an individual's expression of their affective state.
• Figure 2: Framework of personalized affective computing techniques reviewed in this paper.
• Figure 3: A demonstration of the user-specific modeling method. Each unique model $M_i$ is trained on a dataset $D_{I_{i}}$ corresponding to a unique individual $I_i$.
• Figure 4: A demonstration of the group-specific modeling method. Each unique model $M_j$ is trained on a cluster dataset $D_{C_{j}}$. Each dataset is formed of a group of datasets $D_{I_{i}}$ corresponding to individuals $I_{i}$ clustered together.
• Figure 5: A demonstration of the weighting-based approach. In (a), a distance metric $\alpha_i$ is calculated between each individual dataset $D_{I_i}$ and the target user dataset $D_{I_t}$. All the resulting weighted datasets $\alpha_i D_{I_i}$ are grouped together to form an augmented dataset used to train the final model $M_t$. In (b), individual models $M_i$ are trained on each $D_{I_{i}}$. Because of the unlabeled data provided, each $M_i$ makes a prediction $D_{I_{t}M_{i}Pred Label}$ that is multiplied by a confidence estimate $\alpha_i$ to obtain the Augmented $D_{I_{t} Labeled}$. Finally, the Augmented $D_{I_{t} Labeled}$ and the originally labeled target data $D_{I_{t} Labeled}$ are combined together to train the individual-specific model $M_t$.