Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and Outputs

Cyrus Vachha, Yixiao Kang, Zach Dive, Ashwat Chidambaram, Anik Gupta, Eunice Jun, Bjoern Hartmann

TL;DR

Dreamcrafter presents a VR-based editor for 3D radiance fields that unites real-time direct manipulation with high-level, generative AI editing. Its modular architecture integrates online proxies and offline full-generation modules to manage latency while enabling object-level prompting, sculpting, and 3D generation, powered by diffusion-based models and 2D proxies. A first-use study shows users prefer prompting for rapid object creation but favor sculpting for local control, with proxies aiding composition though lacking precise scale information. The system demonstrates a practical path toward immersive, AI-assisted world-building in radiance-field representations and provides a flexible framework adaptable to future 3D representations and generative models.

Abstract

Authoring 3D scenes is a central task for spatial computing applications. Competing visions for lowering existing barriers are (1) focus on immersive, direct manipulation of 3D content or (2) leverage AI techniques that capture real scenes (3D Radiance Fields such as, NeRFs, 3D Gaussian Splatting) and modify them at a higher level of abstraction, at the cost of high latency. We unify the complementary strengths of these approaches and investigate how to integrate generative AI advances into real-time, immersive 3D Radiance Field editing. We introduce Dreamcrafter, a VR-based 3D scene editing system that: (1) provides a modular architecture to integrate generative AI algorithms; (2) combines different levels of control for creating objects, including natural language and direct manipulation; and (3) introduces proxy representations that support interaction during high-latency operations. We contribute empirical findings on control preferences and discuss how generative AI interfaces beyond text input enhance creativity in scene editing and world building.

Dreamcrafter: Immersive Editing of 3D Radiance Fields Through Flexible, Generative Inputs and Outputs

TL;DR

Dreamcrafter presents a VR-based editor for 3D radiance fields that unites real-time direct manipulation with high-level, generative AI editing. Its modular architecture integrates online proxies and offline full-generation modules to manage latency while enabling object-level prompting, sculpting, and 3D generation, powered by diffusion-based models and 2D proxies. A first-use study shows users prefer prompting for rapid object creation but favor sculpting for local control, with proxies aiding composition though lacking precise scale information. The system demonstrates a practical path toward immersive, AI-assisted world-building in radiance-field representations and provides a flexible framework adaptable to future 3D representations and generative models.

Abstract

Authoring 3D scenes is a central task for spatial computing applications. Competing visions for lowering existing barriers are (1) focus on immersive, direct manipulation of 3D content or (2) leverage AI techniques that capture real scenes (3D Radiance Fields such as, NeRFs, 3D Gaussian Splatting) and modify them at a higher level of abstraction, at the cost of high latency. We unify the complementary strengths of these approaches and investigate how to integrate generative AI advances into real-time, immersive 3D Radiance Field editing. We introduce Dreamcrafter, a VR-based 3D scene editing system that: (1) provides a modular architecture to integrate generative AI algorithms; (2) combines different levels of control for creating objects, including natural language and direct manipulation; and (3) introduces proxy representations that support interaction during high-latency operations. We contribute empirical findings on control preferences and discuss how generative AI interfaces beyond text input enhance creativity in scene editing and world building.
Paper Structure (41 sections, 9 figures, 1 table)

This paper contains 41 sections, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Radiance Fields.
  4. Generative Image Models.
  5. Related Work
  6. Generating and editing novel 3D representations using Radiance Fields
  7. Generative Scene Building Systems and Interfaces
  8. Creation Systems in VR
  9. Design Goals
  10. System Design and Implementation
  11. Key interactions
  12. Move objects
  13. Edit radiance field objects via prompting
  14. Generate objects via prompting
  15. Sculpt then stylize objects
  16. ...and 26 more sections

Figures (9)

  • Figure 1: Dreamcrafter system overview. Modules processing pipeline: The Unity project sends API calls to the broker server to run instructions from specific generation modules and their outputs get sent back to the Unity project. Online modules are run for previewing generations, and offline modules are run after editing is complete.
  • Figure 2: Object transformations and direct manipulations: (Left) Positioning object in the scene (Center) Rotating object. (Right) Scaling object
  • Figure 3: Radiance Field Object Editing with preview: (Left) Edit variants are presented to a user. (Center) Displaying selected edit preview as a spatial annotation. (Right) Fully processed 3D edit replaces the original
  • Figure 4: Object Generation via Prompting: (Left) Object generation variations from speech input. (Center) Displaying selected generation preview as a spatial annotation. (Right) Fully processed 3D generation in the scene.
  • Figure 5: Object Generation via Sculpting: (Left) Sculpting toolkit to create primitive shape arrangement (Center) Displaying stylized sculpted object preview as a spatial annotation. (Right) Fully processed 3D generation in the scene.
  • ...and 4 more figures