DT-NVS: Diffusion Transformers for Novel View Synthesis

TL;DR

DT-NVS tackles single-image novel view synthesis in real-world, unaligned scenes by introducing a 3D-aware diffusion model with a transformer backbone and camera-parameter conditioning. It predicts a radiance field via a Vector-Matrix Representation (VM) and renders novel views through volume rendering, facilitated by a training paradigm that swaps reference and noisy inputs. The approach achieves superior FID and competitive perceptual metrics compared to state-of-the-art deterministic and diffusion baselines on MVImgNet and ShapeNet, while enabling diverse outputs. This work broadens 3D diffusion capabilities to real-world, multi-category data and non-aligned camera setups, with potential impact on scalable 3D content synthesis from casual video collections.

Abstract

Generating novel views of a natural scene, e.g., every-day scenes both indoors and outdoors, from a single view is an under-explored problem, even though it is an organic extension to the object-centric novel view synthesis. Existing diffusion-based approaches focus rather on small camera movements in real scenes or only consider unnatural object-centric scenes, limiting their potential applications in real-world settings. In this paper we move away from these constrained regimes and propose a 3D diffusion model trained with image-only losses on a large-scale dataset of real-world, multi-category, unaligned, and casually acquired videos of everyday scenes. We propose DT-NVS, a 3D-aware diffusion model for generalized novel view synthesis that exploits a transformer-based architecture backbone. We make significant contributions to transformer and self-attention architectures to translate images to 3d representations, and novel camera conditioning strategies to allow training on real-world unaligned datasets. In addition, we introduce a novel training paradigm swapping the role of reference frame between the conditioning image and the sampled noisy input. We evaluate our approach on the 3D task of generalized novel view synthesis from a single input image and show improvements over state-of-the-art 3D aware diffusion models and deterministic approaches, while generating diverse outputs.

Figures (6)

• Figure 1: Architecture: DT-NVS is a 3D-aware diffusion model that takes noise $z^i_t$ at $I_3$ and a reference image $x^r$ at reference viewpoints $R_{ir}, T_{ir}$ as input and learns to denoise from $c^i$ and do novel view synthesis on $R_{in}, T_{in}$. The encoder processes noise $z$ and the reference image $x^r$ separately, and generates feature tokens for each. Our decoder concatenates both feature tokens with its output tokens, then applies self-attention with conditioning tokens on their respective camera parameters (input view for output tokens and feature tokens from $z_i^t$, and reference viewpoints on features tokens from $x^r$) We reshape the output tokens into vector-matrix representation, and perform volume rendering. During the training, we supervise the model with denoising loss at input view $I_3$ and photometric loss on novel view. For inference, we use the predicted output at input view ${\hat{x^i}}$ and denoise according to diffusion step $t$.
• Figure 2: Camera Conditioning: We apply AdaLN separately for output tokens, feature tokens from noisy input image and those from reference image. As input rotation matrices are always identity, so we condition on camera distance $r_d$ and focal length $f$ on output tokens and feature tokens from $z_t^i$ with different embedding MLPs. We condition feature tokens from $x^r$ on relative pose $R_{ir}$, $T_{ir}$ between input camera pose $c^i$ and reference camera pose $c^r$.
• Figure 3: Sampling results on ShapeNet: Given reference images, we show different sampling results at input viewpoint (denoised), e.g., identity pose, and novel view synthesis results for objects of the car and chair category from ShapeNet.
• Figure 4: Qualitative Results on MVImgNet: We show the capabilities of DT-NVS on test images from unknown scenes of MVImgNet yu2023mvimgnet. The model takes a reference image as input and denoises at input viewpoint during sampling. We show novel view synthesis results for three different viewpoints after the denoising step.
• Figure 5: Qualitative Results with depth prediction: We show reference images, generated new views and the generated depth maps for a variety of scenes from MVImgNet, as well as a comparison with the ground truth views.