Novel View Synthesis with Pixel-Space Diffusion Models

TL;DR

This work adapts a modern diffusion model architecture for end-to-end NVS in the pixel space, substantially outperforming previous state-of-the-art (SOTA) techniques and introduces a novel NVS training scheme that utilizes single-view datasets, capitalizing on their relative abundance compared to their multi-view counterparts.

Abstract

Synthesizing a novel view from a single input image is a challenging task. Traditionally, this task was approached by estimating scene depth, warping, and inpainting, with machine learning models enabling parts of the pipeline. More recently, generative models are being increasingly employed in novel view synthesis (NVS), often encompassing the entire end-to-end system. In this work, we adapt a modern diffusion model architecture for end-to-end NVS in the pixel space, substantially outperforming previous state-of-the-art (SOTA) techniques. We explore different ways to encode geometric information into the network. Our experiments show that while these methods may enhance performance, their impact is minor compared to utilizing improved generative models. Moreover, we introduce a novel NVS training scheme that utilizes single-view datasets, capitalizing on their relative abundance compared to their multi-view counterparts. This leads to improved generalization capabilities to scenes with out-of-domain content.

Figures (10)

• Figure 1: Novel view synthesis results from our diffusion model. Source views are taken from RealEstate10K realestate10k, and fed into our base and SR models to produce a $256\times256$-pixel prediction. Our end-to-end system implicitly learns to preserve the features in the source view, transform their position along with the camera movement, and generate realistic details in unseen areas.
• Figure 2: Overview of our model. The decoder (purple) learns to denoise the target view, using information from the source view provided by the encoder (blue) through cross-attention. Both models are aware of the diffusion timestep and scene geometry (green).
• Figure 3: $256\times256$-pixel images from RealEstate10K realestate10k, encoded and decoded using the autoencoder from Stable Diffusion v1.4 rombach2022high. Some areas with severe loss of detail are highlighted.
• Figure 4: We select 3 random points in the target view, and show their epipolar lines in the source view in the corresponding color. In epipolar attention, we add a cross-attention bias relative to the proximity of the source view token to the target epipolar line.
• Figure 5: Different NVS results for the same input sampled from our model. Details from the source view are kept in all samples, and diverse realistic options are generated in newly visible areas.