AI-Driven Stylization of 3D Environments

TL;DR

This approach leverages existing image stylization systems and image-to-3D generative models to create a pipeline that iteratively stylizes and composites 3D objects into scenes.

Abstract

In this system, we discuss methods to stylize a scene of 3D primitive objects into a higher fidelity 3D scene using novel 3D representations like NeRFs and 3D Gaussian Splatting. Our approach leverages existing image stylization systems and image-to-3D generative models to create a pipeline that iteratively stylizes and composites 3D objects into scenes. We show our results on adding generated objects into a scene and discuss limitations.

Figures (6)

• Figure 1: Result from InstructPix2Pix, with text prompt: a modern bed in the apartment, clean background
• Figure 2: We combine the generated meshes from GRM in the Unity Scene
• Figure 3: Comparison of apartment scene before and after adding objects using our pipeline.
• Figure 4: Results from each step in our pipeline before SIGNeRF.
• Figure 5: Three objects generated from our pipeline viewed at different view angles.