AI-based Verbal and Visual Scaffolding in a Serious Game: Effects on Learning and Cognitive Load

TL;DR

This study evaluates three AI-mediated scaffolding designs in a quantum-education serious game (no scaffolding, verbal AI, and verbal+visual AI) with 152 participants. All groups show significant learning gains, but only the verbal+visual condition reduces intrinsic cognitive load compared with verbal-only scaffolding, suggesting visual demonstrations ease processing while preserving active problem solving. AI interactions predominantly address level-related questions and action recommendations, indicating the NPC serves as a co-learner rather than a solution provider. The findings underscore the importance of aligning AI capabilities with game design to enable effective, cognitively engaging support without encouraging offloading of effort to AI.

Abstract

Due to their interactive nature, serious games offer valuable opportunities for supporting learning in educational contexts. Recent advances in large language models (LLMs) have further opened the door to new forms of personalized scaffolding in education. In this study, we combine both worlds and study three types of AI-based scaffolding designs in a serious game: (i) no scaffolding, (ii) chat-based (verbal) scaffolding provided by an AI-based non-player character (NPC), and (iii) combined chat-(verbal) and action-based (visual) scaffolding in which the AI may both try to explain or demonstrate the next step towards a solution. The scaffolding conditions are embedded in Qookies, a serious game designed to introduce fundamental concepts of quantum technologies. A total of 152 school students, university students, and members of the general public were randomly assigned to one of the three conditions. The results show that all groups experience significant learning gains, confirming the overall effectiveness of the serious game itself. No significant differences in learning outcomes emerged between scaffolding conditions. However, intrinsic cognitive load was lower in the combined chat-and-action (verbal+visual) scaffolding condition compared to the chat (verbal)-only condition, suggesting that visual demonstrations may offer more accessible support. Interaction analyses further revealed that players engaged with the AI character primarily for level-related questions and action recommendations, while deeper interactions were relatively rare.

Figures (5)

• Figure 1: Example of an interactive educational puzzle in Qookies. Learners progress through the game by manipulating visual elements and solving level-specific challenges that embody core concepts of quantum technologies.
• Figure 2: Brief learning unit providing concise background information on a quantum technology concept. These units introduce theoretical ideas in simple and accessible language and are directly connected to the interactive puzzles.
• Figure 3: AI-based non-player character (NPC) providing learning support in Qookies. Learners can request verbal explanations via chat through the "Talk to" menu or trigger visual (action-based) scaffolding through the "Do something" menu, allowing the AI to demonstrate potentially helpful in-game actions.
• Figure 4: Pre- and post-test scores in conceptual understanding (maximum score = 16) by group (control: no scaffolding; verbal scaffolding: chat-based; verbal and visual scaffolding: chat- and action-based). Asterisks indicate statistical significance: *** $p$ < 0.001, ** $p$ < 0.01, * $p$ < 0.05.
• Figure 5: Self-reported intrinsic, extraneous, and germane cognitive load by group. Cognitive load was assessed using items adapted from leppink2013 (see \ref{['app:CL_questionnaire']}) on a 0–6 Likert scale (low to high cognitive load). Groups differed in scaffolding: control (no AI support), verbal scaffolding (chat-based AI), and verbal and visual scaffolding (chat- and action-based AI). Asterisks indicate statistical significance: *** $p$ < 0.001, ** $p$ < 0.01, * $p$ < 0.05.