NVS-Solver: Video Diffusion Model as Zero-Shot Novel View Synthesizer
Meng You, Zhiyu Zhu, Hui Liu, Junhui Hou
TL;DR
NVS-Solver presents a training-free approach to novel view synthesis by steering pre-trained video diffusion models with scene priors derived from warped input views. It introduces adaptive score modulation through a theoretically grounded, error-bound-based lambda to balance diffusion guidance and view-consistency, enabling high-fidelity NVS from single, multi-view, or monocular video inputs. Extensive experiments on static and dynamic scenes demonstrate state-of-the-art performance in both visual quality and pose accuracy, with ablations validating the core components. The method broadens zero-shot capabilities in view synthesis and suggests avenues for integrating diffusion-based generative models with explicit camera-geometry constraints.
Abstract
By harnessing the potent generative capabilities of pre-trained large video diffusion models, we propose NVS-Solver, a new novel view synthesis (NVS) paradigm that operates \textit{without} the need for training. NVS-Solver adaptively modulates the diffusion sampling process with the given views to enable the creation of remarkable visual experiences from single or multiple views of static scenes or monocular videos of dynamic scenes. Specifically, built upon our theoretical modeling, we iteratively modulate the score function with the given scene priors represented with warped input views to control the video diffusion process. Moreover, by theoretically exploring the boundary of the estimation error, we achieve the modulation in an adaptive fashion according to the view pose and the number of diffusion steps. Extensive evaluations on both static and dynamic scenes substantiate the significant superiority of our NVS-Solver over state-of-the-art methods both quantitatively and qualitatively. \textit{ Source code in } \href{https://github.com/ZHU-Zhiyu/NVS_Solver}{https://github.com/ZHU-Zhiyu/NVS$\_$Solver}.