Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CheapNVS: Real-Time On-Device Narrow-Baseline Novel View Synthesis

Konstantinos Georgiadis, Mehmet Kerim Yucel, Albert Saa-Garriga

TL;DR

CheapNVS tackles single-view NVS under narrow baselines by reframing warping and occlusion filling as concurrent, learnable processes. It introduces a two-encoder, three-decoder architecture that shares a latent representation and uses a pose-conditioned warping module to enable parallel warping and inpainting, trained in multiple stages on large-scale Open Images data. The method achieves faster-than-state-of-the-art inference, lower memory usage, and real-time deployment on mobile devices, while maintaining competitive accuracy across Open Images and COCO test sets. Limitations include warping domain gaps for larger baselines, presenting avenues for future improvements in baseline generalization and inpainting guidance.

Abstract

Single-view novel view synthesis (NVS) is a notorious problem due to its ill-posed nature, and often requires large, computationally expensive approaches to produce tangible results. In this paper, we propose CheapNVS: a fully end-to-end approach for narrow baseline single-view NVS based on a novel, efficient multiple encoder/decoder design trained in a multi-stage fashion. CheapNVS first approximates the laborious 3D image warping with lightweight learnable modules that are conditioned on the camera pose embeddings of the target view, and then performs inpainting on the occluded regions in parallel to achieve significant performance gains. Once trained on a subset of Open Images dataset, CheapNVS outperforms the state-of-the-art despite being 10 times faster and consuming 6% less memory. Furthermore, CheapNVS runs comfortably in real-time on mobile devices, reaching over 30 FPS on a Samsung Tab 9+.

CheapNVS: Real-Time On-Device Narrow-Baseline Novel View Synthesis

TL;DR

CheapNVS tackles single-view NVS under narrow baselines by reframing warping and occlusion filling as concurrent, learnable processes. It introduces a two-encoder, three-decoder architecture that shares a latent representation and uses a pose-conditioned warping module to enable parallel warping and inpainting, trained in multiple stages on large-scale Open Images data. The method achieves faster-than-state-of-the-art inference, lower memory usage, and real-time deployment on mobile devices, while maintaining competitive accuracy across Open Images and COCO test sets. Limitations include warping domain gaps for larger baselines, presenting avenues for future improvements in baseline generalization and inpainting guidance.

Abstract

Single-view novel view synthesis (NVS) is a notorious problem due to its ill-posed nature, and often requires large, computationally expensive approaches to produce tangible results. In this paper, we propose CheapNVS: a fully end-to-end approach for narrow baseline single-view NVS based on a novel, efficient multiple encoder/decoder design trained in a multi-stage fashion. CheapNVS first approximates the laborious 3D image warping with lightweight learnable modules that are conditioned on the camera pose embeddings of the target view, and then performs inpainting on the occluded regions in parallel to achieve significant performance gains. Once trained on a subset of Open Images dataset, CheapNVS outperforms the state-of-the-art despite being 10 times faster and consuming 6% less memory. Furthermore, CheapNVS runs comfortably in real-time on mobile devices, reaching over 30 FPS on a Samsung Tab 9+.
Paper Structure (14 sections, 5 equations, 2 figures, 2 tables)

This paper contains 14 sections, 5 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. CheapNVS
  4. Preliminaries
  5. A better NVS formulation.
  6. CheapNVS
  7. The architecture
  8. Multi-stage training
  9. Experimental Results
  10. Experimental Details
  11. Results
  12. Ablations
  13. Limitations and Future Work
  14. Conclusion

Figures (2)

  • Figure 1: Our CheapNVS architecture. CheapNVS embeds target camera pose and RGBD input information into a shared latent space, which is then used by flow and mask decoders to perform learnable warping, and inpainting decoder to fill in occluded areas. Performing inpainting and warping in parallel, as well as approximating 3D warping via flow and mask decoders make CheapNVS more computationally friendly than existing methods.
  • Figure 2: Left to right: Input, ground-truth occlusion mask, CheapNVS occlusion mask, ground-truth warped RGB, CheapNVS warped RGB, AdaMPI inpainting, CheapNVS inpainting. CheapNVS manages to handle object border artefacts in inpainting (1st, 3rd and 4th rows) and approximates warping successfully.