3D-free meets 3D priors: Novel View Synthesis from a Single Image with Pretrained Diffusion Guidance

TL;DR

The paper tackles the challenge of generating camera-controlled novel views from a single image without relying on large 3D training data. It introduces a hybrid approach that fuses off-the-shelf 3D priors with a 3D-free diffusion-based inference-time optimization, enriching the CLIP space with angle information via a target embedding. A four-step process refines text embeddings and LoRA-enabled diffusion networks on both the input image $I_{input}$ and a weakly guided view $I_{view}$, supplemented by a viewpoint regularization loss $L_{reg}=\| e_{view}- e_{target} \|^2$ to align with target angles $(\alpha_{elev}, \alpha_{azi})$. Experiments on HawkI-Syn and HawkI-Real show substantial improvements over both 3D-based and 3D-free baselines in fidelity, background preservation, and viewpoint accuracy, demonstrating a data-efficient path to high-quality, controllable NVS in complex scenes.

Abstract

Recent 3D novel view synthesis (NVS) methods often require extensive 3D data for training, and also typically lack generalization beyond the training distribution. Moreover, they tend to be object centric and struggle with complex and intricate scenes. Conversely, 3D-free methods can generate text-controlled views of complex, in-the-wild scenes using a pretrained stable diffusion model without the need for a large amount of 3D-based training data, but lack camera control. In this paper, we introduce a method capable of generating camera-controlled viewpoints from a single input image, by combining the benefits of 3D-free and 3D-based approaches. Our method excels in handling complex and diverse scenes without extensive training or additional 3D and multiview data. It leverages widely available pretrained NVS models for weak guidance, integrating this knowledge into a 3D-free view synthesis style approach, along with enriching the CLIP vision-language space with 3D camera angle information, to achieve the desired results. Experimental results demonstrate that our method outperforms existing models in both qualitative and quantitative evaluations, achieving high-fidelity, consistent novel view synthesis at desired camera angles across a wide variety of scenes while maintaining accurate, natural detail representation and image clarity across various viewpoints. We also support our method with a comprehensive analysis of 2D image generation models and the 3D space, providing a solid foundation and rationale for our solution.

Figures (13)

• Figure 1: Our model is capable of generating high quality camera-controlled images at specific azimuth and elevation angles for a variety of complex scenes, all without requiring extra 3D datasets or extensive training. The image in the bottom right corner showcases the output from the 3D-based baseline, Zero123++ shi2023zero123++, created from a designated angle.
• Figure 2: Method. Our method generates a high fidelity camera controlled novel viewpoint of a single image $I_{input}$, its text description and designated angle information. It infuses prior information from pre-trained NVS models into the text to image stable diffusion architecture in a 3D-free inference-time optimization procedure.
• Figure 3: Analysis of how well CLIP understands the 3D space In this experiment, we generate camera control images for specific angles without using any guidance image.
• Figure 4: Using an image with an incorrect viewpoint as the guidance image In this experiment, we examine how the results are derived when an incorrect viewpoint image is used as a guidance image.
• Figure 5: Results on HawkI-Syn. Comparisons between the state-of-the-art view synthesis models, Zero123++, HawkI, Stable Zero123, and our method highlights the superior performance of our model in terms of background inclusion, view consistency, and the accurate representation of target elevation and azimuth angles.