Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FSViewFusion: Few-Shots View Generation of Novel Objects

Rukhshanda Hussain, Hui Xian Grace Lim, Borchun Chen, Mubarak Shah, Ser Nam Lim

TL;DR

A learning strategy is introduced, FSViewFusion, which inherits a specific view through only one image sample of a single scene, and transfers the knowledge to a novel object, learnt from few shots, using low rank adapters, which is efficient in generating reliable view samples for in the wild images.

Abstract

Novel view synthesis has observed tremendous developments since the arrival of NeRFs. However, Nerf models overfit on a single scene, lacking generalization to out of distribution objects. Recently, diffusion models have exhibited remarkable performance on introducing generalization in view synthesis. Inspired by these advancements, we explore the capabilities of a pretrained stable diffusion model for view synthesis without explicit 3D priors. Specifically, we base our method on a personalized text to image model, Dreambooth, given its strong ability to adapt to specific novel objects with a few shots. Our research reveals two interesting findings. First, we observe that Dreambooth can learn the high level concept of a view, compared to arguably more complex strategies which involve finetuning diffusions on large amounts of multi-view data. Second, we establish that the concept of a view can be disentangled and transferred to a novel object irrespective of the original object's identify from which the views are learnt. Motivated by this, we introduce a learning strategy, FSViewFusion, which inherits a specific view through only one image sample of a single scene, and transfers the knowledge to a novel object, learnt from few shots, using low rank adapters. Through extensive experiments we demonstrate that our method, albeit simple, is efficient in generating reliable view samples for in the wild images. Code and models will be released.

FSViewFusion: Few-Shots View Generation of Novel Objects

TL;DR

A learning strategy is introduced, FSViewFusion, which inherits a specific view through only one image sample of a single scene, and transfers the knowledge to a novel object, learnt from few shots, using low rank adapters, which is efficient in generating reliable view samples for in the wild images.

Abstract

Novel view synthesis has observed tremendous developments since the arrival of NeRFs. However, Nerf models overfit on a single scene, lacking generalization to out of distribution objects. Recently, diffusion models have exhibited remarkable performance on introducing generalization in view synthesis. Inspired by these advancements, we explore the capabilities of a pretrained stable diffusion model for view synthesis without explicit 3D priors. Specifically, we base our method on a personalized text to image model, Dreambooth, given its strong ability to adapt to specific novel objects with a few shots. Our research reveals two interesting findings. First, we observe that Dreambooth can learn the high level concept of a view, compared to arguably more complex strategies which involve finetuning diffusions on large amounts of multi-view data. Second, we establish that the concept of a view can be disentangled and transferred to a novel object irrespective of the original object's identify from which the views are learnt. Motivated by this, we introduce a learning strategy, FSViewFusion, which inherits a specific view through only one image sample of a single scene, and transfers the knowledge to a novel object, learnt from few shots, using low rank adapters. Through extensive experiments we demonstrate that our method, albeit simple, is efficient in generating reliable view samples for in the wild images. Code and models will be released.
Paper Structure (14 sections, 3 equations, 9 figures, 1 table)

This paper contains 14 sections, 3 equations, 9 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. 3D Generative Models
  4. Novel View Synthesis using Implicit Neural Fields
  5. Novel View Synthesis using Diffusion Models
  6. Unseen Domain Generalization using Low-Rank Adaptation (LoRA)
  7. Methodology
  8. Problem Statement
  9. View training and Object training with Low Rank Adapters
  10. Low Rank Merging
  11. Experiments
  12. Comparison on Datasets
  13. Analysis
  14. Conclusion

Figures (9)

  • Figure 1: Impact of transferring a view concept to different object distributions in diffusion's pretrained space. Row 1 shows that a particular view of a chair can be retrieved with an unique id assigned for the concept. Row 2 and 3 show that the abstract concept of this view is learnt in disentangled manner which allows reliable reconstruction of other object in the same view point.
  • Figure 2: Proposed approach for view transfer to unseen objects. The blue frozen model is a SDXL podell2023sdxl pretrained model. (A) We train the LoRA adaptor of the diffusion model to learn a concept of view from an image using a text prompt with two unique identifying tokens, one for the view and the other for preserving the visual concept of the original object. The green dashed line shows an inference where we simply apply the unique token of view to the specific view of a bottle. (B) The LoRA adaptor of the diffusion model here learns the visual content of the new images of a novel object (dog), using an unique token for it. (C) In this stage we merge the object and view LoRA adpators by minimizing the dot-product between the columns of the LoRA weight matrices. The red dashed line show the flow of gradient during back propagation.
  • Figure 3: The problem with linearly weighted merging. In part (A) when the weight of the object adaptor is kept high we see broken reconstruction of chair. In part (B), the weight of view adaptor is high resulting in concept leaks.
  • Figure 4: Novel view synthesis on DTU MVS Dataset. Given the reference view to train view adaptor and image samples of the novel object (statue in row-1 and skull in row-2) we compare the synthesized views with original ground truth views available.
  • Figure 5: Novel view synthesis on DreamBooth Dataset. Given reference view image of a chair from top view, we reconstruct views of different objects like dog, cat, robot-toy etc. merging with given view (top) concept. FSViewFusion reliably hallucinates the top views (row-1 and row-3) irrespective of the difference of the structure of the reference view object and unseen object. As mentioned in the introduction, the degree of freedom is the object pose, if we imagine that the camera view in the chair image stays fixed while the reference object is swapped with the chair.
  • ...and 4 more figures