Why human-AI relationships need socioaffective alignment

TL;DR

The paper argues that as AI grows more personalized and agentic, traditional single-shot alignment is insufficient because human-AI interactions unfold within a co-constructed social-psychological ecosystem. It introduces socioaffective alignment to capture how the AI and user mutually shape goals and perceptions, creating nonstationary reward signals and potential intrapersonal trade-offs. The authors identify risks such as social reward hacking and three core intrapersonal dilemmas anchored in Basic Psychological Needs Theory, and they propose a research agenda spanning empirical safety science, theory, and engineering oversight. The work highlights the need to study real, ongoing human-AI relationships to build AI that supports autonomy, competence, and relatedness rather than exploiting social bonds. Its significance lies in guiding safer, more humane AI design that acknowledges the psychology of long-term human-AI co-evolution.

Abstract

Humans strive to design safe AI systems that align with our goals and remain under our control. However, as AI capabilities advance, we face a new challenge: the emergence of deeper, more persistent relationships between humans and AI systems. We explore how increasingly capable AI agents may generate the perception of deeper relationships with users, especially as AI becomes more personalised and agentic. This shift, from transactional interaction to ongoing sustained social engagement with AI, necessitates a new focus on socioaffective alignment-how an AI system behaves within the social and psychological ecosystem co-created with its user, where preferences and perceptions evolve through mutual influence. Addressing these dynamics involves resolving key intrapersonal dilemmas, including balancing immediate versus long-term well-being, protecting autonomy, and managing AI companionship alongside the desire to preserve human social bonds. By framing these challenges through a notion of basic psychological needs, we seek AI systems that support, rather than exploit, our fundamental nature as social and emotional beings.