LLM Theory of Mind and Alignment: Opportunities and Risks

TL;DR

This paper investigates how theory of mind (ToM)—the ability to infer others' mental states—renders LLMs capable of more nuanced user interactions and group coordination. It maps potential ToM manifestations across individual and group contexts, detailing opportunities to improve goal specification, conversational adaptation, empathy, and collective alignment, alongside risks of manipulation, deception, privacy loss, and pathological attachments. It argues that higher-order ToM can both enhance and threaten alignment, depending on context and governance, and emphasizes the need for empirical validation and normative guardrails. The work aims to guide the design and governance of LLMs as they participate in multi-user social domains, balancing improved human–AI communication with safeguards against misuse.

Abstract

Large language models (LLMs) are transforming human-computer interaction and conceptions of artificial intelligence (AI) with their impressive capacities for conversing and reasoning in natural language. There is growing interest in whether LLMs have theory of mind (ToM); the ability to reason about the mental and emotional states of others that is core to human social intelligence. As LLMs are integrated into the fabric of our personal, professional and social lives and given greater agency to make decisions with real-world consequences, there is a critical need to understand how they can be aligned with human values. ToM seems to be a promising direction of inquiry in this regard. Following the literature on the role and impacts of human ToM, this paper identifies key areas in which LLM ToM will show up in human:LLM interactions at individual and group levels, and what opportunities and risks for alignment are raised in each. On the individual level, the paper considers how LLM ToM might manifest in goal specification, conversational adaptation, empathy and anthropomorphism. On the group level, it considers how LLM ToM might facilitate collective alignment, cooperation or competition, and moral judgement-making. The paper lays out a broad spectrum of potential implications and suggests the most pressing areas for future research.