LLMER: Crafting Interactive Extended Reality Worlds with JSON Data Generated by Large Language Models

TL;DR

LLMER addresses the challenge of building interactive XR worlds without brittle script generation by converting natural language input into structured JSON data generated by large language models. It introduces a multi-stage framework powered by a Context Library and an LLM Wrapper, plus three execution modules (Virtual Object Creator, Animation Library, Reality Fusion Engine) to realize objects and interactions in Unity-powered XR scenes. A preliminary user study shows substantial efficiency gains, including approximately 80% token reduction and around 60% faster task completion relative to code-based baselines, along with generally positive usability feedback. The work demonstrates a practical, open-source pathway to robust, responsive XR experiences driven by JSON data rather than programmatic scripts, with clear directions for memory, object fidelity, and cross-platform scalability improvements.

Abstract

The integration of Large Language Models (LLMs) like GPT-4 with Extended Reality (XR) technologies offers the potential to build truly immersive XR environments that interact with human users through natural language, e.g., generating and animating 3D scenes from audio inputs. However, the complexity of XR environments makes it difficult to accurately extract relevant contextual data and scene/object parameters from an overwhelming volume of XR artifacts. It leads to not only increased costs with pay-per-use models, but also elevated levels of generation errors. Moreover, existing approaches focusing on coding script generation are often prone to generation errors, resulting in flawed or invalid scripts, application crashes, and ultimately a degraded user experience. To overcome these challenges, we introduce LLMER, a novel framework that creates interactive XR worlds using JSON data generated by LLMs. Unlike prior approaches focusing on coding script generation, LLMER translates natural language inputs into JSON data, significantly reducing the likelihood of application crashes and processing latency. It employs a multi-stage strategy to supply only the essential contextual information adapted to the user's request and features multiple modules designed for various XR tasks. Our preliminary user study reveals the effectiveness of the proposed system, with over 80% reduction in consumed tokens and around 60% reduction in task completion time compared to state-of-the-art approaches. The analysis of users' feedback also illuminates a series of directions for further optimization.

Figures (10)

• Figure 1: System architecture of LLMER. In the figure, blue, black, and orange lines represent the transmission of audio files, text strings, and structured JSON data, respectively. LLMER records user audio and utilizes the Whisper Model to transcribe it into text. The text-based user requests are then processed by the LLM Wrapper using a multi-stage approach, during which LLMER interacts with the GPT-4 model through text-based inputs and outputs. Additionally, the output from the LLM Wrapper can be parsed into structured JSON data or conversational text. The JSON data is used to either retrieve contextual information from the Context Library or execute various XR tasks through three designated modules, while the conversational text is converted into synthetic speech by a text-to-speech model to emulate human-like conversation.
• Figure 2: Example usage of the Context Library. Based on the JSON data included in the initial response, the Context Library provides essential data to be integrated into the refined prompt.
• Figure 3: A request sent to the Virtual Object Creator. For requests related to creating virtual objects, the refined response is supposed to include JSON data following specific JSON schema.
• Figure 4: Examples of creating complex objects using local resources.
• Figure 5: A request sent to the Animation Library. By parsing the action types and corresponding properties, the Animation Library generates various types of animations in a fast and reliable way.