Warmth and competence in human-agent cooperation

TL;DR

This work shows that when humans cooperate with AI agents in a mixed-motive setting, social perceptions—especially perceived warmth—predict preferences for agents beyond what objective performance reveals. By training deep RL agents with a Social Value Orientation and a trembling-hand mechanism in Coins, the authors quantify how warmth and competence shape both stated and revealed preferences across three studies (N=501). Results consistently show warmth as a robust predictor, with competence having more mixed or negative effects, and scores providing incremental but weaker predictive power. The findings argue for integrating social perception metrics into human-agent interaction research to better forecast adoption, trust, and ongoing collaboration with AI systems.

Abstract

Interaction and cooperation with humans are overarching aspirations of artificial intelligence (AI) research. Recent studies demonstrate that AI agents trained with deep reinforcement learning are capable of collaborating with humans. These studies primarily evaluate human compatibility through "objective" metrics such as task performance, obscuring potential variation in the levels of trust and subjective preference that different agents garner. To better understand the factors shaping subjective preferences in human-agent cooperation, we train deep reinforcement learning agents in Coins, a two-player social dilemma. We recruit $N = 501$ participants for a human-agent cooperation study and measure their impressions of the agents they encounter. Participants' perceptions of warmth and competence predict their stated preferences for different agents, above and beyond objective performance metrics. Drawing inspiration from social science and biology research, we subsequently implement a new ``partner choice'' framework to elicit revealed preferences: after playing an episode with an agent, participants are asked whether they would like to play the next episode with the same agent or to play alone. As with stated preferences, social perception better predicts participants' revealed preferences than does objective performance. Given these results, we recommend human-agent interaction researchers routinely incorporate the measurement of social perception and subjective preferences into their studies.

Figures (41)

• Figure 1: When humans encounter a new agent, they automatically and rapidly form an impression of the agent (social perception). The human can leverage their impression to decide whether to continue or discontinue the interaction (partner choice).
• Figure 2: Screenshot of gameplay in Coins. Two players move around a small room and collect colored coins. Coins randomly appear in the room over time. Players receive reward from coin collections depending on the match or mismatch between their color and the coin color.
• Figure 3: Questionnaires administered in the human-agent interaction studies.
• Figure 4: Performance metrics over agent training. Selfish agents quickly learned to collect coins, but did not learn to avoid mismatches. As a result, collective return hovered around zero. Prosocial agents exhibited slower learning and collected fewer coins on average, but also learned to avoid mismatching coins. As a result, collective return increased markedly over training. Error bands represent 95% confidence intervals over 100 evaluation episodes run at regular training checkpoints.
• Figure 5: Main effects of algorithmic components on social perceptions in Study 1. (a) An agent's Social Value Orientation (SVO) significantly influenced perceived warmth, ${p < 0.001}$. (b) Similarly, the trembling-hand component significantly changed competence judgments, ${p < 0.001}$. Error bars indicate 95% confidence intervals.