Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MagicItem: Dynamic Behavior Design of Virtual Objects with Large Language Models in a Consumer Metaverse Platform

Ryutaro Kurai, Takefumi Hiraki, Yuichi Hiroi, Yutaro Hirao, Monica Perusquia-Hernandez, Hideaki Uchiyama, Kiyoshi Kiyokawa

TL;DR

This work tackles the barrier non-programmers face in defining VR object behaviors by introducing MagicItem, a system that translates natural language into Cluster Script within the Cluster metaverse platform. It integrates GPT-4-based code generation with a dedicated web server and uses TypeScript definition files to ground the LLM in the Cluster Script API, enabling real-time testing and synchronization across users. A large-scale online study with 63 participants demonstrates that many non-programmers can successfully generate object behaviors, with novices reporting higher usability (SUS around 62) and the experiment highlighting expressiveness in Task 3 through creative, language-driven prompts. The paper contributes a practical system design, prompt-engineering strategy, and empirical evaluation while outlining limitations and directions for more accessible, collaborative, and advanced VR content creation in the metaverse.

Abstract

To create rich experiences in virtual reality (VR) environments, it is essential to define the behavior of virtual objects through programming. However, programming in 3D spaces requires a wide range of background knowledge and programming skills. Although Large Language Models (LLMs) have provided programming support, they are still primarily aimed at programmers. In metaverse platforms, where many users inhabit VR spaces, most users are unfamiliar with programming, making it difficult for them to modify the behavior of objects in the VR environment easily. Existing LLM-based script generation methods for VR spaces require multiple lengthy iterations to implement the desired behaviors and are difficult to integrate into the operation of metaverse platforms. To address this issue, we propose a tool that generates behaviors for objects in VR spaces from natural language within Cluster, a metaverse platform with a large user base. By integrating LLMs with the Cluster Script provided by this platform, we enable users with limited programming experience to define object behaviors within the platform freely. We have also integrated our tool into a commercial metaverse platform and are conducting online experiments with 63 general users of the platform. The experiments show that even users with no programming background can successfully generate behaviors for objects in VR spaces, resulting in a highly satisfying system. Our research contributes to democratizing VR content creation by enabling non-programmers to design dynamic behaviors for virtual objects in metaverse platforms.

MagicItem: Dynamic Behavior Design of Virtual Objects with Large Language Models in a Consumer Metaverse Platform

TL;DR

This work tackles the barrier non-programmers face in defining VR object behaviors by introducing MagicItem, a system that translates natural language into Cluster Script within the Cluster metaverse platform. It integrates GPT-4-based code generation with a dedicated web server and uses TypeScript definition files to ground the LLM in the Cluster Script API, enabling real-time testing and synchronization across users. A large-scale online study with 63 participants demonstrates that many non-programmers can successfully generate object behaviors, with novices reporting higher usability (SUS around 62) and the experiment highlighting expressiveness in Task 3 through creative, language-driven prompts. The paper contributes a practical system design, prompt-engineering strategy, and empirical evaluation while outlining limitations and directions for more accessible, collaborative, and advanced VR content creation in the metaverse.

Abstract

To create rich experiences in virtual reality (VR) environments, it is essential to define the behavior of virtual objects through programming. However, programming in 3D spaces requires a wide range of background knowledge and programming skills. Although Large Language Models (LLMs) have provided programming support, they are still primarily aimed at programmers. In metaverse platforms, where many users inhabit VR spaces, most users are unfamiliar with programming, making it difficult for them to modify the behavior of objects in the VR environment easily. Existing LLM-based script generation methods for VR spaces require multiple lengthy iterations to implement the desired behaviors and are difficult to integrate into the operation of metaverse platforms. To address this issue, we propose a tool that generates behaviors for objects in VR spaces from natural language within Cluster, a metaverse platform with a large user base. By integrating LLMs with the Cluster Script provided by this platform, we enable users with limited programming experience to define object behaviors within the platform freely. We have also integrated our tool into a commercial metaverse platform and are conducting online experiments with 63 general users of the platform. The experiments show that even users with no programming background can successfully generate behaviors for objects in VR spaces, resulting in a highly satisfying system. Our research contributes to democratizing VR content creation by enabling non-programmers to design dynamic behaviors for virtual objects in metaverse platforms.
Paper Structure (49 sections, 5 figures, 1 table)

This paper contains 49 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Social VR Platforms
  4. AI-Assisted 3D Space Design and Control
  5. Large Language Models for Code Generation
  6. MagicItem
  7. System Design
  8. Prompt Engineering
  9. Selection of LLMs for Code Generation
  10. User Study
  11. Participant Demographics
  12. Pre-experiment Setup
  13. Tasks and Metrics
  14. Tasks
  15. Task 1:
  16. ...and 34 more sections

Figures (5)

  • Figure 1: The goal image of Task 1: an avatar jumping higher than usual.
  • Figure 2: The goal image of Task 2: a white box floating in a position without ground, which is normally unreachable, and an avatar sitting on top of it.
  • Figure 3: Completion time for each task for participants who are inexperienced and experienced Cluster Script.
  • Figure 4: The figure (A) shows the number of trials for each participant, (B) shows the time taken to generate scripts, (C) shows the length of the instructions entered by the participants to generate the Cluster Scripts, and (D) measures the length by the quantity of tokens in GPT-4-turbo.
  • Figure 5: Scores for each of the six fundamental items used to assess task workload in the NASA-TLX.