Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and Gemini

Ruofei Du, Benjamin Hersh, David Li, Nels Numan, Xun Qian, Yanhe Chen, Zhongyi Zhou, Xingyue Chen, Jiahao Ren, Robert Timothy Bettridge, Steve Toh, David Kim

Abstract

While large language models have accelerated software development through "vibe coding", prototyping intelligent Extended Reality (XR) experiences remains inaccessible due to the friction of complex game engines and low-level sensor integration. To bridge this gap, we contribute XR Blocks, an open-source, modular WebXR framework that abstracts spatial computing complexities into high-level, human-centered primitives. Building upon this foundation, we present Vibe Coding XR, an end-to-end rapid prototyping workflow that leverages LLMs to translate natural language intent directly into functional XR software. Using a web-based interface, creators can transform high-level prompts (e.g., "create a dandelion that reacts to hand") into interactive WebXR applications in under a minute. We provide a preliminary technical evaluation on a pilot dataset (VCXR60) alongside diverse application scenarios highlighting mixed-reality realism, multi-modal interaction, and generative AI integrations. By democratizing spatial software creation, this work empowers practitioners to bypass low-level hurdles and rapidly move from "idea to reality." Code and live demos are available at https://xrblocks.github.io/gem and https://github.com/google/xrblocks.

Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and Gemini

Abstract

While large language models have accelerated software development through "vibe coding", prototyping intelligent Extended Reality (XR) experiences remains inaccessible due to the friction of complex game engines and low-level sensor integration. To bridge this gap, we contribute XR Blocks, an open-source, modular WebXR framework that abstracts spatial computing complexities into high-level, human-centered primitives. Building upon this foundation, we present Vibe Coding XR, an end-to-end rapid prototyping workflow that leverages LLMs to translate natural language intent directly into functional XR software. Using a web-based interface, creators can transform high-level prompts (e.g., "create a dandelion that reacts to hand") into interactive WebXR applications in under a minute. We provide a preliminary technical evaluation on a pilot dataset (VCXR60) alongside diverse application scenarios highlighting mixed-reality realism, multi-modal interaction, and generative AI integrations. By democratizing spatial software creation, this work empowers practitioners to bypass low-level hurdles and rapidly move from "idea to reality." Code and live demos are available at https://xrblocks.github.io/gem and https://github.com/google/xrblocks.

Paper Structure

This paper contains 17 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. GenXR: Creating AI + XR Experiences
  4. Vibe Coding: Gaps from AI to XR
  5. Vibe Coding XR
  6. XR Blocks Framework
  7. Vibe Coding XR Workflow
  8. Application Scenarios: From Prompt to XR
  9. Educational AI + XR Experiences
  10. Physics-Aware XR Realism & Interaction
  11. Rapid Game Prototyping & Procedural Generation
  12. Preliminary Technical Evaluation
  13. Limitation and Discussion
  14. Conclusion
  15. VCXR60 Dataset Prompts
  16. ...and 2 more sections

Figures (5)

  • Figure 1: Vibe Coding XR accelerates AI + XR prototyping by allowing users to (A) test their "vibe coding" results on desktop in a "simulated reality" environment, and (B) deploy the same demo on an Android XR headset with body and hand interactions.
  • Figure 2: Design of the XR Blocks Framework: (A) conceptual abstraction of the "Reality Model", and (B) modular architecture of the "core" engine. Subsystems marked with ${}^\ast$ have not yet been fully open sourced.
  • Figure 3: Human-coded templates and samples in the XR Blocks framework provide the foundational best practices and API grounding for Vibe Coding XR.
  • Figure 4: AI-generated application for educational and exercise use cases. See \ref{['sec:application_scnarios_prompts']} for full prompts.
  • Figure :