MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenes

TL;DR

MOVIS tackles the challenge of multi-object novel-view synthesis in indoor scenes by enriching diffusion-based view-conditioned models with structure-aware inputs (depth and object masks) and an auxiliary novel-view mask prediction task. It introduces a structure-guided timestep scheduler to balance learning global object placement and fine-grained details, and proposes new metrics to assess cross-view consistency. Through extensive experiments on the synthetic C3DFS dataset and generalization tests on Room-Texture, Objaverse, 3D-FRONT, and SUNRGB-D, MOVIS demonstrates improved object placement, geometry, and appearance across novel views and strong cross-dataset generalization. The work provides practical insights for designing 3D-aware multi-object NVS systems and offers a scalable framework for future research in compositional scene synthesis.

Abstract

Repurposing pre-trained diffusion models has been proven to be effective for NVS. However, these methods are mostly limited to a single object; directly applying such methods to compositional multi-object scenarios yields inferior results, especially incorrect object placement and inconsistent shape and appearance under novel views. How to enhance and systematically evaluate the cross-view consistency of such models remains under-explored. To address this issue, we propose MOVIS to enhance the structural awareness of the view-conditioned diffusion model for multi-object NVS in terms of model inputs, auxiliary tasks, and training strategy. First, we inject structure-aware features, including depth and object mask, into the denoising U-Net to enhance the model's comprehension of object instances and their spatial relationships. Second, we introduce an auxiliary task requiring the model to simultaneously predict novel view object masks, further improving the model's capability in differentiating and placing objects. Finally, we conduct an in-depth analysis of the diffusion sampling process and carefully devise a structure-guided timestep sampling scheduler during training, which balances the learning of global object placement and fine-grained detail recovery. To systematically evaluate the plausibility of synthesized images, we propose to assess cross-view consistency and novel view object placement alongside existing image-level NVS metrics. Extensive experiments on challenging synthetic and realistic datasets demonstrate that our method exhibits strong generalization capabilities and produces consistent novel view synthesis, highlighting its potential to guide future 3D-aware multi-object NVS tasks. Our project page is available at https://jason-aplp.github.io/MOVIS/.

Figures (17)

• Figure 1: Overview of MOVIS. Our model performs nvs from the input image and relative camera change. We introduce structure-aware features as additional inputs and employ mask prediction as an auxiliary task (\ref{['sec:method_structure_aware']}). The model is trained with a structure-guided timestep sampling scheduler (\ref{['sec:method_scheduler']}) to balance the learning of global object placement and local detail recovery.
• Figure 2: Visualization of inference. The early stage of the denoising process focuses on restoring global object placements, while the prediction of object masks requires a relatively noiseless image to recover fine-grained geometry. This motivates us to seek a balanced timestep sampling scheduler during training. The model trained w/ shift yields better mask prediction and thus recovers an image with more details and sharp object boundary. The w/o shift here refers to not shifting the $\mu$ value.
• Figure 3: Qualitative results of nvs and cross-view matching. Our method generates plausible novel-view images across various datasets, surpassing baselines regarding object placement, shape, and appearance. In cross-view matching, points of the same color indicate correspondences between the input and target views. We achieve a higher number of matched points with more precise locations.
• Figure 4: Qualitative comparison for ablation study. Excluding mask predictions or the scheduler reduces the model’s ability to learn object placement, as shown by the brown cabinet example.
• Figure S.5: Illustration of different timestep sampling strategies.