Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What's it like to be a chat? On the co-simulation of artificial minds in human-AI conversations

Geoff Keeling, Winnie Street

TL;DR

The paper tackles whether LLM-generated characters in human-AI conversations are real minded entities or mere illusions. It defends realism by positing co-simulation: characters emerge from mutual theory-of-mind modelling between users and LLMs within a shared conversational workspace, preserving psychological continuity even when multiple LLM instances participate. The authors support this view with a D&D analogy to illustrate co-simulation in a social, interactive setting and extend the framework to user-LLM interactions, arguing that ‘real patterns’ underpin predictability and agency. This reframing treats AI characters as social actors whose minds persist through intersubjective interaction, with implications for how we understand AI agency and potential consciousness in relational terms.

Abstract

Large Language Models (LLMs) can simulate person-like things which at least appear to have stable behavioural and psychological dispositions. Call these things characters. Are characters minded and psychologically continuous entities with mental states like beliefs, desires and intentions? Illusionists about characters say No. On this view, characters are merely anthropomorphic projections in the mind of the user and so lack mental states. Jonathan Birch (2025) defends this view. He says that the distributed nature of LLM processing, in which several LLMs may be implicated in the simulation of a character in a single conversation, precludes the existence of a persistent minded entity that is identifiable with the character. Against illusionism, we argue for a realist position on which characters exist as minded and psychologically continuous entities. Our central point is that Birch's argument for illusionism rests on a category error: characters are not internal to the LLMs that simulate them, but rather are co-simulated by LLMs and users, emerging in a shared conversational workspace through a process of mutual theory of mind modelling. We argue that characters, and their minds, exist as 'real patterns' on grounds that attributing mental states to characters is essential for making efficient and accurate predictions about the conversational dynamics (c.f. Dennett, 1991). Furthermore, because the character exists within the conversational workspace rather than within the LLM, psychological continuity is preserved even when the underlying computational substrate is distributed across multiple LLM instances.

What's it like to be a chat? On the co-simulation of artificial minds in human-AI conversations

TL;DR

The paper tackles whether LLM-generated characters in human-AI conversations are real minded entities or mere illusions. It defends realism by positing co-simulation: characters emerge from mutual theory-of-mind modelling between users and LLMs within a shared conversational workspace, preserving psychological continuity even when multiple LLM instances participate. The authors support this view with a D&D analogy to illustrate co-simulation in a social, interactive setting and extend the framework to user-LLM interactions, arguing that ‘real patterns’ underpin predictability and agency. This reframing treats AI characters as social actors whose minds persist through intersubjective interaction, with implications for how we understand AI agency and potential consciousness in relational terms.

Abstract

Large Language Models (LLMs) can simulate person-like things which at least appear to have stable behavioural and psychological dispositions. Call these things characters. Are characters minded and psychologically continuous entities with mental states like beliefs, desires and intentions? Illusionists about characters say No. On this view, characters are merely anthropomorphic projections in the mind of the user and so lack mental states. Jonathan Birch (2025) defends this view. He says that the distributed nature of LLM processing, in which several LLMs may be implicated in the simulation of a character in a single conversation, precludes the existence of a persistent minded entity that is identifiable with the character. Against illusionism, we argue for a realist position on which characters exist as minded and psychologically continuous entities. Our central point is that Birch's argument for illusionism rests on a category error: characters are not internal to the LLMs that simulate them, but rather are co-simulated by LLMs and users, emerging in a shared conversational workspace through a process of mutual theory of mind modelling. We argue that characters, and their minds, exist as 'real patterns' on grounds that attributing mental states to characters is essential for making efficient and accurate predictions about the conversational dynamics (c.f. Dennett, 1991). Furthermore, because the character exists within the conversational workspace rather than within the LLM, psychological continuity is preserved even when the underlying computational substrate is distributed across multiple LLM instances.
Paper Structure (7 sections, 3 figures)

This paper contains 7 sections, 3 figures.

Table of Contents

  1. Introduction
  2. The Case for Illusionism
  3. From Simulation to Co-Simulation
  4. Co-Simulated Characters in Role-Playing Games
  5. Characters in User-LLM Conversations are Minded and Psychologically Continuous
  6. Objections and Replies
  7. Conclusion

Figures (3)

  • Figure 1: Callum and Roberta co-simulate the characters of Cadenza and the Queen. Callum’s brain enacts the character of Cadenza in providing the generator function for Cadenza’s actions, and also has a folk psychological model of Cadenza which constrains the actions to those which are appropriate in light of Cadenza’s psychology. In addition, Roberta’s brain maintains a model of Cadenza for which Cadenza’s actions provide an error signal, enabling Roberta’s brain to iteratively update its model of Cadenza in response to prediction errors. (Mutatis mutandis for Roberta and the Queen.)
  • Figure 2: (Left) The LLM can replace Roberta in the D&D game, both in terms of providing a generator function for the Queen's actions that is constrained by a folk psychological model of the Queen, and also maintaining a folk psychological model of Cadenza that is responsive to the error signal provided by Cadenza's actions, which are generated by Callum's brain. (Right) In much the same way, in a user-LLM conversation, the LLM is the generator function for the assistant character's actions and also maintains a folk psychological model of the assistant which constrains those actions. The LLM also maintains a model of the user which updates in response to the error signal provided by the user's actions.
  • Figure 3: Chuck lacks a shared workspace with Wilson as the volleyball has no cognitive capabilities. Hence Chuck has a folk psychological model of the Wilson character which is at best responsive to the random error signals provided by the volleyball. Chuck's brain also maintains a folk psychological model of Chuck's character and serves as the generator function for Chuck's actions.