ImaginateAR: AI-Assisted In-Situ Authoring in Augmented Reality

TL;DR

ImaginateAR tackles the challenge of making personalized AR authoring accessible to non-experts in outdoor environments. It introduces an in-situ, speech-driven workflow that combines offline outdoor scene understanding, fast 3D asset generation, and LLM-based reasoning to place and arrange virtual content on real-world scenes. Key contributions include an updated outdoor scene understanding pipeline built on OpenMask3D plus GPT-4o labeling and clustering, a fast remote asset-generation pipeline using DALL-E 2 and InstantMesh, and a multi-agent LLM system for brainstorming, planning, and assembly. User studies show that participants favor hybrid AI-human authoring for speed and creativity while preferring manual control for precision, highlighting implications for design of accessible, controllable AI-assisted AR tools.

Abstract

While augmented reality (AR) enables new ways to play, tell stories, and explore ideas rooted in the physical world, authoring personalized AR content remains difficult for non-experts, often requiring professional tools and time. Prior systems have explored AI-driven XR design but typically rely on manually defined VR environments and fixed asset libraries, limiting creative flexibility and real-world relevance. We introduce ImaginateAR, the first mobile tool for AI-assisted AR authoring to combine offline scene understanding, fast 3D asset generation, and LLMs -- enabling users to create outdoor scenes through natural language interaction. For example, saying "a dragon enjoying a campfire" (P7) prompts the system to generate and arrange relevant assets, which can then be refined manually. Our technical evaluation shows that our custom pipelines produce more accurate outdoor scene graphs and generate 3D meshes faster than prior methods. A three-part user study (N=20) revealed preferred roles for AI, how users create in freeform use, and design implications for future AR authoring tools. ImaginateAR takes a step toward empowering anyone to create AR experiences anywhere -- simply by speaking their imagination.

Figures (11)

• Figure 1: Diagram of the 3D scene understanding pipeline. Given an input point cloud, we first estimate 3D masks. Next, we assign a semantic label to each mask using a VLM and propagate the label to all points within the mask, producing a semantic point cloud. We then cluster nearby points with the same label to infer the final set of 3D masks, from which we extract 3D bounding boxes. For visualization, we show only the bounding boxes, not the underlying masks. The Pavement box is enclosed within the Road box and is therefore not visible.
• Figure 2: Results of the 3D scene understanding module. For each of the three scans---Vase, House, and Garden---we visualize the input point cloud (left) and the final set of labeled 3D bounding boxes inferred by our scene understanding pipeline (right). We also report the total time (in minutes) required to estimate the scene graph for each scan. Note that some bounding boxes may be enclosed within others and may therefore be occluded.
• Figure 3: Example of 3D asset generation. Given a user prompt, we first apply prompt boosting, then use Dall-E 2 ramesh2022hierarchical to generate a consistent image by editing the center region of a white canvas. The image is then lifted to 3D using InstantMesh xu2024instantmesh. The 'Bad" example (right) illustrates a failure case because it would produce a partial 3D object (i.e., only the dragon’s head). Prompt boosting helps avoid such incomplete generations.
• Figure 4: Different screen captures of the ImaginateAR's mobile interface showing the UI layout and functionalities. Users can access manual, AI-assisted, and AI-decided modes across different features through buttons on the screen.
• Figure 5: From left to right: bounding boxes from the ground truth, OpenMask3D takmaz2023openmask3d, and our proposed method. OpenMask3D predicts a large number of masks, resulting in excessive bounding boxes that over-represent the same scene objects. In contrast, our method produces fewer, more accurate boxes. (Box colors are arbitrary and can be ignored.)