DORSal: Diffusion for Object-centric Representations of Scenes et al

TL;DR

DORSal addresses scalable, high-quality 3D novel-view synthesis across diverse scenes by conditioning a diffusion-based decoder on frozen object-centric scene representations (OSRT). By using Object Slot conditioning and per-view camera poses, the method yields 3D-consistent renderings with object-level editability and improved perceptual metrics over prior scene-representation baselines. Experiments on MultiShapeNet and Street View show DORSal achieves better FID and LPIPS scores and supports object removal and transfer between scenes, while maintaining view consistency along camera paths. The work demonstrates how diffusion models can leverage structured object-centric conditioning for controllable 3D scene generation, with potential for end-to-end training and large-scale video-data integration.

Abstract

Recent progress in 3D scene understanding enables scalable learning of representations across large datasets of diverse scenes. As a consequence, generalization to unseen scenes and objects, rendering novel views from just a single or a handful of input images, and controllable scene generation that supports editing, is now possible. However, training jointly on a large number of scenes typically compromises rendering quality when compared to single-scene optimized models such as NeRFs. In this paper, we leverage recent progress in diffusion models to equip 3D scene representation learning models with the ability to render high-fidelity novel views, while retaining benefits such as object-level scene editing to a large degree. In particular, we propose DORSal, which adapts a video diffusion architecture for 3D scene generation conditioned on frozen object-centric slot-based representations of scenes. On both complex synthetic multi-object scenes and on the real-world large-scale Street View dataset, we show that DORSal enables scalable neural rendering of 3D scenes with object-level editing and improves upon existing approaches.

Figures (14)

• Figure 1: Model overview. (a) OSRT is trained to predict novel views through an Encoder-Decoder architecture with an Object Slot latent representation of the scene. Since the model is trained with the L2 loss and the task contains significant amounts of ambiguity, the predictions are commonly blurry. (b) After training the OSRT model, and freezing it, we take the Object Slots and combine it with the target Poses to be used as conditioning. Our Multiview U-Net is trained in a diffusion process to denoise novel views while cross-attending into the conditioning features (see Figure \ref{['fig:dorsal_detail']} for details). This results in sharp renders at test time, which can still be decomposed into the objects in the scene to support edits.
• Figure 2: DORSal slot and pose conditioning. DORSal is conditioned via cross-attention and FiLM-modulation perez2018film on a set of Object Slots (shared across views) and a per-view Pose vector.
• Figure 3: DORSal scene editing and evaluation. To obtain instance segmentations of objects in a scene, we perform scene edits by dropping out individual slots, rendering the resulting views, and computing a pixel-wise difference (middle) compared to the unedited rendered views (left). These differences are smoothed and thresholded to arrive at a segmentation image (right).
• Figure 4: Novel View Synthesis. Comparison of DORSal with the following baselines: 3DiM watson2023novel, SRT srt, and OSRT osrt on the MultiShapeNet (only 2/5 views shown) and Street View datasets.
• Figure 5: Scene editing: object transfer. We highlight several transferred objects. Note that transferred objects are rendered consistently across views (see circled objects in final row) while taking into account global illumination properties of the scene in which they are placed in (e.g. shadows are rendered correctly for transferred objects).