Real-Time Position-Aware View Synthesis from Single-View Input

TL;DR

This work addresses real-time synthesis of novel views from a single image given a target pose, a challenging problem due to depth ambiguity and latency requirements. It introduces PVSNet, which combines a compact 6D pose embedding with a dual-encoder rendering network to produce high-fidelity views at real-time speeds (≈55 FPS at 512×512, and ≈126 FPS for narrow baselines in LF contexts). The approach demonstrates strong quantitative and qualitative performance on Blender and COCO datasets, and provides competitive light-field reconstruction capabilities, while maintaining low latency suitable for telepresence and AR applications. The results are reinforced by thorough ablations, revealing the importance of the normalization+positional encoding+MLP pipeline and a dual-encoder structure for robust, pose-aware view synthesis from minimal input.

Abstract

Recent advancements in view synthesis have significantly enhanced immersive experiences across various computer graphics and multimedia applications, including telepresence and entertainment. By enabling the generation of new perspectives from a single input view, view synthesis allows users to better perceive and interact with their environment. However, many state-of-the-art methods, while achieving high visual quality, face limitations in real-time performance, which makes them less suitable for live applications where low latency is critical. In this paper, we present a lightweight, position-aware network designed for real-time view synthesis from a single input image and a target camera pose. The proposed framework consists of a Position Aware Embedding, which efficiently maps positional information from the target pose to generate high dimensional feature maps. These feature maps, along with the input image, are fed into a Rendering Network that merges features from dual encoder branches to resolve both high and low level details, producing a realistic new view of the scene. Experimental results demonstrate that our method achieves superior efficiency and visual quality compared to existing approaches, particularly in handling complex translational movements without explicit geometric operations like warping. This work marks a step toward enabling real-time live and interactive telepresence applications.

Figures (8)

• Figure 1: Pipeline of view synthesis using PVSNet which takes single input image $I_s$ & target location $P_t$, and produces the image $\hat{I}_t$. The Positional embedding branch takes $P_t$ and creates a higher dimensional projection of positional feature $\mathbf{\varrho}_{H \times W \times C}$, which then along with input image $I_s$ is passed to rendering network responsible for producing output image $\hat{I}_t$.
• Figure 2: Conceptual view of camera placement of each stage of training. Stage 1 involves small baseline and a fixed grid of cameras, stage 2 focuses on larger baseline but still keep the discrete placement of cameras on a fixed grid. Finally, stage 3 trains the network on large baseline with random camera placements in a cubical volume. The gray volume roughly describes a 20 $cm^3$ volume. Bright colors on the rightmost figure are closer to us and darker ones are away.
• Figure 3: Some example images taken from each unique scene from the blender dataset.
• Figure 4: Quality vs FPS rate of different methods against different resolution for Blender dataset.
• Figure 5: Comparison of view synthesis results on COCO dataset with error maps for SinMPI pu2023sinmpi, TMPI khan2023tiled_tmpi, AdaMPI han2022single_AdaMPI, and our method. The top and bottom rows depict the synthesized views for two different scenes, with corresponding error maps underneath. The red rectangles highlight specific regions of interest to emphasize differences in synthesis accuracy and visual fidelity.