Novel View Extrapolation with Video Diffusion Priors

TL;DR

ViewExtrapolator is a generic novel view extrapolator that can work with different types of 3D rendering such as views rendered from point clouds when only a single view or monocular video is available and requires no fine-tuning of SVD, making it both data-efficient and computation-efficient.

Abstract

The field of novel view synthesis has made significant strides thanks to the development of radiance field methods. However, most radiance field techniques are far better at novel view interpolation than novel view extrapolation where the synthesis novel views are far beyond the observed training views. We design ViewExtrapolator, a novel view synthesis approach that leverages the generative priors of Stable Video Diffusion (SVD) for realistic novel view extrapolation. By redesigning the SVD denoising process, ViewExtrapolator refines the artifact-prone views rendered by radiance fields, greatly enhancing the clarity and realism of the synthesized novel views. ViewExtrapolator is a generic novel view extrapolator that can work with different types of 3D rendering such as views rendered from point clouds when only a single view or monocular video is available. Additionally, ViewExtrapolator requires no fine-tuning of SVD, making it both data-efficient and computation-efficient. Extensive experiments demonstrate the superiority of ViewExtrapolator in novel view extrapolation. Project page: \url{https://kunhao-liu.github.io/ViewExtrapolator/}.

Figures (10)

• Figure 1: We introduce ViewExtrapolator, a novel approach that leverages the generative priors of Stable Video Diffusion for novel view extrapolation, where the novel views lie far beyond the range of the training views. ViewExtrapolator effectively refines the artifact-prone renderings (left side of arrows) of radiance fields or point clouds, to more realistic renderings with fewer artifacts (right side of arrows).
• Figure 2: The setting differences between novel view interpolation and novel view extrapolation: Radiance fields excel at novel view interpolation but struggle at novel view extrapolation.
• Figure 3: Overview of the proposed ViewExtrapolator. We render an artifact-prone video from the closest training view to an extrapolative novel view, and then refine it by guiding SVD to preserve the original scene content and eliminate the artifacts with guidance annealing and resampling annealing.
• Figure 4: Qualitative comparisons. We compare ViewExtrapolator with 3DGS and DRGS on novel view extrapolation. ViewExtrapolator demonstrates superior generation quality with much fewer artifacts. The last column shows the distribution of training and test views as well as the corresponding extrapolation degree $e$. Zoom in for details.
• Figure 5: The definition of extrapolation degree$e$ by the ratio between $\mathbf{d}$ and $r$ ($\mathbf{d}$ stands for the distance between the novel view and the central point of training views, and $r$ stands for the training view range as the maximum extent of the training views along the direction of $\mathbf{d}$). A higher $e$ means that the novel view is farther away from the training views.