VR-GPT: Visual Language Model for Intelligent Virtual Reality Applications

TL;DR

VR-GPT addresses the need for natural, task-aware guidance in immersive VR by integrating a Visual Language Model with a Unity-based VR platform and speech interfaces. It delivers real-time, audio-based instructions without on-screen text and is trained on VR-specific scene data. Evaluations show improved user engagement and reduced task completion times and errors in kitchen and laboratory tasks, across both familiar and unfamiliar scenarios. The work demonstrates a practical path for deploying VLM reasoning in interactive XR applications with potential for real-world utility.

Abstract

The advent of immersive Virtual Reality applications has transformed various domains, yet their integration with advanced artificial intelligence technologies like Visual Language Models remains underexplored. This study introduces a pioneering approach utilizing VLMs within VR environments to enhance user interaction and task efficiency. Leveraging the Unity engine and a custom-developed VLM, our system facilitates real-time, intuitive user interactions through natural language processing, without relying on visual text instructions. The incorporation of speech-to-text and text-to-speech technologies allows for seamless communication between the user and the VLM, enabling the system to guide users through complex tasks effectively. Preliminary experimental results indicate that utilizing VLMs not only reduces task completion times but also improves user comfort and task engagement compared to traditional VR interaction methods.

Figures (3)

• Figure 1: Application of VLM-VR integration system to the medical task.
• Figure 2: System Architecture.
• Figure 3: Performance comparison for two categories of tasks for different environments.