Set-the-Scene: Global-Local Training for Generating Controllable NeRF Scenes

TL;DR

<3-5 sentence high-level summary> Set-the-Scene addresses the lack of controllability in text-to-3D NeRFs by introducing a composable framework where each object is an independent NeRF anchored to an explicit proxy. It employs a Global-Local training strategy that alternates between local object optimization and global scene optimization, both guided by a score-distillation loss $L_{sds}$ derived from a pretrained diffusion prior $\mathcal{M}$. The method supports multiple proxies per NeRF, proxy-driven geometry via a shape proxy with loss $L_{shape}$, and post-training editing of placement, geometry, and color without restarting from scratch. Compared with single-object baselines like Latent-NeRF and DreamFusion, it shows improved scene coherence, object compatibility, and editability, enabling interactive, text-guided 3D scene synthesis with composable control.

Abstract

Recent breakthroughs in text-guided image generation have led to remarkable progress in the field of 3D synthesis from text. By optimizing neural radiance fields (NeRF) directly from text, recent methods are able to produce remarkable results. Yet, these methods are limited in their control of each object's placement or appearance, as they represent the scene as a whole. This can be a major issue in scenarios that require refining or manipulating objects in the scene. To remedy this deficit, we propose a novel GlobalLocal training framework for synthesizing a 3D scene using object proxies. A proxy represents the object's placement in the generated scene and optionally defines its coarse geometry. The key to our approach is to represent each object as an independent NeRF. We alternate between optimizing each NeRF on its own and as part of the full scene. Thus, a complete representation of each object can be learned, while also creating a harmonious scene with style and lighting match. We show that using proxies allows a wide variety of editing options, such as adjusting the placement of each independent object, removing objects from a scene, or refining an object. Our results show that Set-the-Scene offers a powerful solution for scene synthesis and manipulation, filling a crucial gap in controllable text-to-3D synthesis.

Figures (13)

• Figure 1: Set-the-Scene allows generating composable and controllable scenes from text prompts and 3D object proxies. (left) The scene is represented using a set of proxies, defining the location, coarse shape, and text prompt of each target object. A set of NeRFs are then optimized with respect to the object proxies and an additional scene text prompt. (right) By manipulating the proxies, the scene can be edited without additional fine-tuning.
• Figure 2: Importance of Global-Local training. (Input) The proxy objects used to define the scene. (Local Only) A scene where each object is optimized only for itself; notice how objects look pasted and do not match in terms of color scheme. (Global-Local) Our global-local training, which also interleaves global training steps of the entire scene.
• Figure 3: Set-the-Scene Training pipeline. A scene is first defined using a set of proxies, where each proxy is coupled with a location and a text prompt. Given an input scene, we then apply a global-local training procedure where we alternate between locally optimizing each object on its own and optimizing the entire scene rendered together as a whole.
• Figure 4: Set-the-Scene results. The same proxy setting can be used to create different styles of the same scene. The scene prompts are shown for each generated scene, and corresponding prompts are used for each object. For example "a kid bedroom style wardrobe, closed doors" or "a baroque chair".
• Figure 5: Scene generation results. Our method is able to handle complex scenes with multiple repeating objects.