Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Social Conjuring: Multi-User Runtime Collaboration with AI in Building Virtual 3D Worlds

Amina Kobenova, Cyan DeVeaux, Samyak Parajuli, Andrzej Banburski-Fahey, Judith Amores Fernandez, Jaron Lanier

TL;DR

Social Conjurer presents a real-time, AI-augmented framework for multi-user co-creation of VR 3D worlds. By integrating LLMs and VLMs with runtime asset generation, object layout, and tool-enabled interactions in Unity, it enables spontaneous collaborative worldbuilding driven by prompts and sketches. A mixed-method study with 12 participants reveals how social contexts shape prompting, the role of AI interpretation quirks, and identifies design implications for accessibility, ethics, and future social-AI VR tools. The work demonstrates a proof-of-concept for post-symbolic, social VR, offering practical pathways for scalable AI-assisted 3D content creation in shared virtual spaces.

Abstract

Generative artificial intelligence has shown promise in prompting virtual worlds into existence, yet little attention has been given to understanding how this process unfolds as social interaction. We present Social Conjurer, a framework for AI-augmented dynamic 3D scene co-creation, where multiple users collaboratively build and modify virtual worlds in real-time. Through an expanded set of interactions, including social and tool-based engagements as well as spatial reasoning, our framework facilitates the creation of rich, diverse virtual environments. Findings from a preliminary user study (N=12) provide insight into the user experience of this approach, how social contexts shape the prompting of spatial environments, and perspective on social applications of prompt-based 3D co-creation. In addition to highlighting the potential of AI-supported multi-user world creation and offering new pathways for AI-augmented creative processes in VR, this article presents a set of implications for designing human-centered interfaces that incorporate AI models into 3D content generation.

Social Conjuring: Multi-User Runtime Collaboration with AI in Building Virtual 3D Worlds

TL;DR

Social Conjurer presents a real-time, AI-augmented framework for multi-user co-creation of VR 3D worlds. By integrating LLMs and VLMs with runtime asset generation, object layout, and tool-enabled interactions in Unity, it enables spontaneous collaborative worldbuilding driven by prompts and sketches. A mixed-method study with 12 participants reveals how social contexts shape prompting, the role of AI interpretation quirks, and identifies design implications for accessibility, ethics, and future social-AI VR tools. The work demonstrates a proof-of-concept for post-symbolic, social VR, offering practical pathways for scalable AI-assisted 3D content creation in shared virtual spaces.

Abstract

Generative artificial intelligence has shown promise in prompting virtual worlds into existence, yet little attention has been given to understanding how this process unfolds as social interaction. We present Social Conjurer, a framework for AI-augmented dynamic 3D scene co-creation, where multiple users collaboratively build and modify virtual worlds in real-time. Through an expanded set of interactions, including social and tool-based engagements as well as spatial reasoning, our framework facilitates the creation of rich, diverse virtual environments. Findings from a preliminary user study (N=12) provide insight into the user experience of this approach, how social contexts shape the prompting of spatial environments, and perspective on social applications of prompt-based 3D co-creation. In addition to highlighting the potential of AI-supported multi-user world creation and offering new pathways for AI-augmented creative processes in VR, this article presents a set of implications for designing human-centered interfaces that incorporate AI models into 3D content generation.
Paper Structure (74 sections, 17 figures, 3 tables)

This paper contains 74 sections, 17 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Creating 3D Content with AI
  4. Spatial Reasoning and Scene Layouts
  5. Collaborative Worldbuilding in 3D and VR
  6. Design Goals
  7. AI-Augmented Creation of Virtual Scenes at Run-Time
  8. Embodied Interaction with Scenes in VR
  9. Virtual Spaces for Collaborative Worldbuilding
  10. System Description
  11. Scene Generator: Spatial Reasoning and Environment Generation
  12. Environment Generation
  13. Asset Proposal and Retrieval
  14. Layout Arrangement
  15. Layout Improvement
  16. ...and 59 more sections

Figures (17)

  • Figure 1: Overview of the single-player Conjurer architecture
  • Figure 2: Examples of scenes generated by Conjurer.
  • Figure 3: High level overview of Social Conjurer. Several of the submodules are themselves multi-agent architectures, described in following subsections.
  • Figure 4: Environment Generation Module for the Social Conjurer System.
  • Figure 5: Examples of environments created with our environment generation module.
  • ...and 12 more figures