Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VRCopilot: Authoring 3D Layouts with Generative AI Models in VR

Lei Zhang, Jin Pan, Jacob Gettig, Steve Oney, Anhong Guo

TL;DR

VRCopilot tackles immersive authoring in VR by integrating pre-trained generative models to co-create indoor layouts. It introduces multimodal specification and intermediate wireframe representations to improve controllability and support rapid prototyping. Two user studies show that scaffolded wireframes enhance user agency, while manual multimodal creation offers the highest sense of creativity; automatic creation yields lower creativity and agency. The findings highlight design opportunities for transparent AI assistance, multiple alternatives, and intermediate representations to balance creativity and control in VR-based design tasks.

Abstract

Immersive authoring provides an intuitive medium for users to create 3D scenes via direct manipulation in Virtual Reality (VR). Recent advances in generative AI have enabled the automatic creation of realistic 3D layouts. However, it is unclear how capabilities of generative AI can be used in immersive authoring to support fluid interactions, user agency, and creativity. We introduce VRCopilot, a mixed-initiative system that integrates pre-trained generative AI models into immersive authoring to facilitate human-AI co-creation in VR. VRCopilot presents multimodal interactions to support rapid prototyping and iterations with AI, and intermediate representations such as wireframes to augment user controllability over the created content. Through a series of user studies, we evaluated the potential and challenges in manual, scaffolded, and automatic creation in immersive authoring. We found that scaffolded creation using wireframes enhanced the user agency compared to automatic creation. We also found that manual creation via multimodal specification offers the highest sense of creativity and agency.

VRCopilot: Authoring 3D Layouts with Generative AI Models in VR

TL;DR

VRCopilot tackles immersive authoring in VR by integrating pre-trained generative models to co-create indoor layouts. It introduces multimodal specification and intermediate wireframe representations to improve controllability and support rapid prototyping. Two user studies show that scaffolded wireframes enhance user agency, while manual multimodal creation offers the highest sense of creativity; automatic creation yields lower creativity and agency. The findings highlight design opportunities for transparent AI assistance, multiple alternatives, and intermediate representations to balance creativity and control in VR-based design tasks.

Abstract

Immersive authoring provides an intuitive medium for users to create 3D scenes via direct manipulation in Virtual Reality (VR). Recent advances in generative AI have enabled the automatic creation of realistic 3D layouts. However, it is unclear how capabilities of generative AI can be used in immersive authoring to support fluid interactions, user agency, and creativity. We introduce VRCopilot, a mixed-initiative system that integrates pre-trained generative AI models into immersive authoring to facilitate human-AI co-creation in VR. VRCopilot presents multimodal interactions to support rapid prototyping and iterations with AI, and intermediate representations such as wireframes to augment user controllability over the created content. Through a series of user studies, we evaluated the potential and challenges in manual, scaffolded, and automatic creation in immersive authoring. We found that scaffolded creation using wireframes enhanced the user agency compared to automatic creation. We also found that manual creation via multimodal specification offers the highest sense of creativity and agency.
Paper Structure (45 sections, 5 figures)

This paper contains 45 sections, 5 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generative Models for 3D Scenes
  4. Creativity Support via Steering Generative Models
  5. Interaction Techniques in Immersive Environments
  6. VRCopilot
  7. Scope
  8. Immersive Authoring Features
  9. Importing Models
  10. Design Exploration
  11. Generative Model in VRCopilot
  12. Multimodal Specification
  13. Intermediate Representation
  14. Ways of Human-AI Co-creation in VR
  15. Manual Creation
  16. ...and 30 more sections

Figures (5)

  • Figure 1: User interfaces in VRCopilot including (a) a palette menu where users can select and furniture from the catalog, and (b) a multi-workspace visualization that allows users to work and switch between multiple versions.
  • Figure 2: VRCopilot proposes three ways of human-AI co-creation in virtual immersive environments: manual creation (1a-e), automatic creation (2a-e), and scaffolded creation (3a-e).
  • Figure 3: The system architecture of VRCopilot.
  • Figure 4: Exemplary top-down view comparison of participants' creation results with and without the assistance of generative AI in Study 1.
  • Figure 5: Results from post-task survey comparing three conditions in Study 2.