Exploring LLM-Powered Role and Action-Switching Pedagogical Agents for History Education in Virtual Reality

TL;DR

This work tackles how to enable engaging, adaptive multi-role pedagogical agents in VR history education using large language models. A Pavilion of Prince Teng VR prototype employs adaptive role-switching and action-switching modules, evaluated through a 2×2 between-subject study with 84 participants to assess learning outcomes and learning experience. Results show adaptive role-switching enhances trustworthiness and expertise, while adaptive action-switching boosts social presence and humanness; neither module significantly affects usability, motivation, or cognitive load, and frequent switching can yield inconsistent experiences. The authors propose five design implications to guide future VR history education tools employing adaptive role-switching and action-switching in multi-role PAs.

Abstract

Multi-role pedagogical agents can create engaging and immersive learning experiences, helping learners better understand knowledge in history learning. However, existing pedagogical agents often struggle with multi-role interactions due to complex controls, limited feedback forms, and difficulty dynamically adapting to user inputs. In this study, we developed a VR prototype with LLM-powered adaptive role-switching and action-switching pedagogical agents to help users learn about the history of the Pavilion of Prince Teng. A 2 x 2 between-subjects study was conducted with 84 participants to assess how adaptive role-switching and action-switching affect participants' learning outcomes and experiences. The results suggest that adaptive role-switching enhances participants' perception of the pedagogical agent's trustworthiness and expertise but may lead to inconsistent learning experiences. Adaptive action-switching increases participants' perceived social presence, expertise, and humanness. The study did not uncover any effects of role-switching and action-switching on usability, learning motivation, and cognitive load. Based on the findings, we proposed five design implications for incorporating adaptive role-switching and action-switching into future VR history education tools.

Figures (8)

• Figure 1: Prototype overview. Prototype structure for VR learning of the Pavilion of Prince Teng with PAs. For adaptive role-switching, the system switches roles based on user input, including voice, tone, and the appearance of the agents. For adaptive action-switching, the system determines and displays appropriate actions based on model output.
• Figure 2: Prototype and different experimental conditions settings. (a) Demonstrates the different experimental group conditions: whether there is adaptive role-switching and adaptive action-switching. (b) Based on the experimental group's requirements, the model will switch between three characters in response to user questions: an Archaeological Expert, Wang Bo (the poet who wrote the "Preface to the Pavilion of Prince Teng"), and Prince Teng (the owner of the Pavilion). Additionally, the model will determine which actions to display based on its output. The action library includes 18 actions, categorized into pointing, natural expressive, descriptive, and character-specific actions.
• Figure 3: Overview of user study procedure. The entire process will take approximately 60 to 80 minutes to complete.
• Figure 4: Prototype user interface. During the learning process, users are provided with text prompts outlining their learning goals related to knowledge of the Pavilion of Prince Teng's architecture and history fact, literary background, and modern interpretations of the Pavilion of Prince Teng on the left side of the interface. Positioned on the right at a 45-degree angle is a PA who continuously accompanies and faces the user. Users can dialogue with the PA by clicking on the buttons on the right side, enabling role and action-switching modules based on different experimental conditions.
• Figure 5: (a) Conceptual score change and (b) factual score change between pre-test and post-test scores. Role-switching and action-switching individually had no significant impact on participants' learning of conceptual knowledge. And role-switching affected participants' learning of factual knowledge.