Evaluating Usability and Engagement of Large Language Models in Virtual Reality for Traditional Scottish Curling

TL;DR

This study evaluates the usability, engagement, and learning impact of integrating large language models (LLMs) into a virtual reality (VR) environment for traditional Scottish curling within the Scottish Bonspiel VR prototype. It compares an LLM-based chatbot against a predefined scripted chatbot in a between-subjects design, examining two VR levels that reflect climate-driven transitions from outdoor to indoor play. The results show significant gains in usability and engagement with the LLM-enabled VR, while learning outcomes show no statistically significant differences, highlighting the potential for LLMs to enhance cultural heritage experiences and accessibility in immersive settings. The findings support broader adoption of VR and LLMs in heritage education to preserve intangible cultural heritage amid environmental changes, while underscoring challenges such as factual accuracy, connectivity, and the need for longitudinal evaluation. Overall, the work demonstrates a scalable, interactive approach to documenting and disseminating heritage knowledge, with implications for engaging younger audiences and advancing experimental preservation in cultural education.

Abstract

This paper explores the innovative application of Large Language Models (LLMs) in Virtual Reality (VR) environments to promote heritage education, focusing on traditional Scottish curling presented in the game ``Scottish Bonspiel VR''. Our study compares the effectiveness of LLM-based chatbots with pre-defined scripted chatbots, evaluating key criteria such as usability, user engagement, and learning outcomes. The results show that LLM-based chatbots significantly improve interactivity and engagement, creating a more dynamic and immersive learning environment. This integration helps document and preserve cultural heritage and enhances dissemination processes, which are crucial for safeguarding intangible cultural heritage (ICH) amid environmental changes. Furthermore, the study highlights the potential of novel technologies in education to provide immersive experiences that foster a deeper appreciation of cultural heritage. These findings support the wider application of LLMs and VR in cultural education to address global challenges and promote sustainable practices to preserve and enhance cultural heritage.

Figures (10)

• Figure 1: Usability evaluation across multiple dimensions: Comparing ChatGPT and predefined chatbot scores. *, **, ***, and **** correspond to the significance levels of $p < .05$, $p < .01$, $p < .001$, and $p < .0001$ respectively. Response options used a Likert scale, spanning from 1 (Not at all agree) to 7 (Very much agree).
• Figure 2: Comparison of User Engagement Between CGPT and PD Conditions. All times are in seconds. Asterisks denote levels of significance where *, **, and *** correspond to the significance levels of $p < .05$, $p < .01$, and $p < .001$ respectively. 'NS' represents no statistical significance.
• Figure 3: The design of the game environment for Level 1 and Level 2.
• Figure 4: Objects in the VR scene are inspired by Scottish curling heritage artifacts.
• Figure 5: The overview of our procedure and experimental setup.