Collaborative Storytelling and LLM: A Linguistic Analysis of Automatically-Generated Role-Playing Game Sessions
Alessandro Maisto
TL;DR
The study investigates whether language produced by Large Language Models in automatically generated RPG sessions exhibits oral or written characteristics by comparing LLM narratives with transcriptions of human RPG play, ordinary conversations, and fantasy books. It uses a multi-LLM setup (one GM and three players) to generate complete sessions and applies a comprehensive linguistic analysis across lexical, syntactic, verbal, and connective features, augmented by Propp's narrative-function mapping. Findings indicate that LLMs display a largely written-like lexical and syntactic profile, with a persistent present-tense bias and notably low cohesion, though their descriptive PC dialogues resemble spoken language more than the GM text. The results suggest LLMs can effectively simulate NPCs and augment RPG narration but currently fall short of capturing the full spontaneity and temporal evolution of human-led games, highlighting directions for training and reinforcement learning to enhance cohesion and affective storytelling.
Abstract
Role-playing games (RPG) are games in which players interact with one another to create narratives. The role of players in the RPG is largely based on the interaction between players and their characters. This emerging form of shared narrative, primarily oral, is receiving increasing attention. In particular, many authors investigated the use of an LLM as an actor in the game. In this paper, we aim to discover to what extent the language of Large Language Models (LLMs) exhibit oral or written features when asked to generate an RPG session without human interference. We will conduct a linguistic analysis of the lexical and syntactic features of the generated texts and compare the results with analyses of conversations, transcripts of human RPG sessions, and books. We found that LLMs exhibit a pattern that is distinct from all other text categories, including oral conversations, human RPG sessions and books. Our analysis has shown how training influences the way LLMs express themselves and provides important indications of the narrative capabilities of these tools.