Towards a Theory of AI Personhood

TL;DR

The paper investigates when AI systems should be considered persons by proposing three necessary conditions—agency, theory-of-mind, and self-awareness—grounded in philosophy and machine learning. It surveys contemporary AI literature to assess whether current systems meet these criteria, finding the evidence mixed and inconclusive. It then analyzes how these personhood facets relate to AI alignment, highlighting both potential benefits (e.g., cooperation via ToM) and risks (e.g., deception, goal-reflection-induced drift). Finally, it outlines open research directions and discusses the ethical and legal implications of AI personhood, arguing that recognizing personhood could fundamentally alter the approach to control and alignment in AI systems.

Abstract

I am a person and so are you. Philosophically we sometimes grant personhood to non-human animals, and entities such as sovereign states or corporations can legally be considered persons. But when, if ever, should we ascribe personhood to AI systems? In this paper, we outline necessary conditions for AI personhood, focusing on agency, theory-of-mind, and self-awareness. We discuss evidence from the machine learning literature regarding the extent to which contemporary AI systems, such as language models, satisfy these conditions, finding the evidence surprisingly inconclusive. If AI systems can be considered persons, then typical framings of AI alignment may be incomplete. Whereas agency has been discussed at length in the literature, other aspects of personhood have been relatively neglected. AI agents are often assumed to pursue fixed goals, but AI persons may be self-aware enough to reflect on their aims, values, and positions in the world and thereby induce their goals to change. We highlight open research directions to advance the understanding of AI personhood and its relevance to alignment. Finally, we reflect on the ethical considerations surrounding the treatment of AI systems. If AI systems are persons, then seeking control and alignment may be ethically untenable.