Playing the Imitation Game: How Perceived Generated Content Shapes Player Experience

TL;DR

This paper investigates how players’ perceptions of whether game levels are AI- or human-generated influence their gameplay experience. Using a mixed-method, Turing-style study across Super Mario Bros. and Sokoban, the authors evaluated 60 levels (15 human, 15 AI per game) with 154 participants, capturing both quantitative experience ratings and qualitative rationales. They find that accuracy in distinguishing creators is near random, but beliefs about origin strongly shape enjoyment, difficulty, frustration, and aesthetics, revealing a pervasive perception bias or 'lemons' dynamic. The work highlights the need for nuanced, contextual disclosures about AI usage in games to preserve trust and guide responsible integration of PCG and GenAI into game design and communities.

Abstract

With the fast progress of generative AI in recent years, more games are integrating generated content, raising questions regarding how players perceive and respond to this content. To investigate, we ran a mixed-method survey on the games Super Mario Bros. and Sokoban, comparing procedurally generated levels and levels designed by humans to explore how perceptions of the creator relate to players' overall experience of gameplay. Players could not reliably identify the level's creator, yet their experiences were strongly linked to their beliefs about that creator rather than the actual truth. Levels believed to be human-made were rated as more fun and aesthetically pleasing. In contrast, those believed to be AI-generated were rated as more frustrating and challenging. This negative bias appeared spontaneously without knowing the levels' creator and often was based on unreliable cues of "human-likeness." Our results underscore the importance of understanding perception biases when integrating generative systems into games.

Figures (12)

• Figure 1: Players have different perceptions about human-likeness. The same game feature (an enemy placed near the player's spawning position) was interpreted as evidence of AI design by one participant and as evidence of human design by another, illustrating how identical cues can support opposite judgments.
• Figure 2: Example screenshot of games.
• Figure 3: Participants' gaming frequency.
• Figure 4: Participants' experience with game design.
• Figure 5: Participants' attitudes toward procedural content generation (PCG) and generative AI (GenAI) in games. Bar charts show the distribution of responses across five categories from "Very negative" to "Very positive."