PsychoGAT: A Novel Psychological Measurement Paradigm through Interactive Fiction Games with LLM Agents

TL;DR

PsychoGAT introduces a multi-agent framework that gamifies standardized psychological assessments by embedding scale items into interactive fiction generated and guided by LLM agents. Through a three-agent design (game designer, game controller, critic) plus a human simulator and psychometric evaluator, the approach aims to achieve robust reliability and validity while improving user engagement and accessibility. Experimental results indicate competitive psychometric metrics and enhanced content quality, supported by both automatic simulations and human evaluations. The work demonstrates the potential of LLM-driven, game-based assessments to broaden reach and acceptance of psychological measurement, while acknowledging the need for longitudinal validation and localization across languages and populations.

Abstract

Psychological measurement is essential for mental health, self-understanding, and personal development. Traditional methods, such as self-report scales and psychologist interviews, often face challenges with engagement and accessibility. While game-based and LLM-based tools have been explored to improve user interest and automate assessment, they struggle to balance engagement with generalizability. In this work, we propose PsychoGAT (Psychological Game AgenTs) to achieve a generic gamification of psychological assessment. The main insight is that powerful LLMs can function both as adept psychologists and innovative game designers. By incorporating LLM agents into designated roles and carefully managing their interactions, PsychoGAT can transform any standardized scales into personalized and engaging interactive fiction games. To validate the proposed method, we conduct psychometric evaluations to assess its effectiveness and employ human evaluators to examine the generated content across various psychological constructs, including depression, cognitive distortions, and personality traits. Results demonstrate that PsychoGAT serves as an effective assessment tool, achieving statistically significant excellence in psychometric metrics such as reliability, convergent validity, and discriminant validity. Moreover, human evaluations confirm PsychoGAT's enhancements in content coherence, interactivity, interest, immersion, and satisfaction.

Figures (9)

• Figure 1: We propose PsychoGAT, a novel psychological assessment paradigm in the form of an interactive game instead of a traditional self-report scale.
• Figure 2: The multi-agent framework of PsychoGAT. The designer generates settings for the interactive fiction game based on a given self-report scale. The controller, critic, and a human participant (or human simulator) engage in a cyclical interaction to facilitate the assessment process. $I^{yes}$ represents the human-selected instruction.
• Figure 3: Comparison of assessment paradigms. Traditional scales and psychologist rolep-playing interviews emphasize the recall of life events and self-reported feelings or thoughts. In contrast, PsychoGAT introduces an interactive fiction game environment where participants make decisions as the protagonist and craft their personal story.
• Figure 4: Comparisons among various assessment methods through automatic and human evaluations. All five methods are qualified for psychometric effectiveness, and PsychoGAT brings all-around user experience improvement. PsyMtrc= Psychometric, CH= Coherence, IA= Interactivity, INT= Interest, IM= Immersion, and ST= Satisfaction.
• Figure 5: Percentage agreements of PsychoGAT's superiority across five human evaluation metrics, as defined in \ref{['fig:baseline_comparison']} and §\ref{['sec: experi_indices']}.