From Rays to Projections: Better Inputs for Feed-Forward View Synthesis

TL;DR

This work tackles the brittleness of input conditioning in feed-forward novel view synthesis caused by Plücker $6$D ray-space representations. It introduces projective conditioning, which replaces absolute camera parameters with a target-view point-cloud projection cue and casts view synthesis as a 2D image-to-image problem, thus improving robustness to camera transformations and geometric consistency. A Masked Auto-Encoding pretraining strategy leverages uncalibrated data to learn a powerful cross-view completion prior within the 2D conditioning framework. Through a dedicated Consistency Benchmark and RealEstate10K-derived data, PVSM achieves state-of-the-art 3D consistency and rendering quality while maintaining competitive runtime, highlighting the practical impact of operating in a gauge-free, 2D-conditioned regime for scalable, robust view synthesis.

Abstract

Feed-forward view synthesis models predict a novel view in a single pass with minimal 3D inductive bias. Existing works encode cameras as Plücker ray maps, which tie predictions to the arbitrary world coordinate gauge and make them sensitive to small camera transformations, thereby undermining geometric consistency. In this paper, we ask what inputs best condition a model for robust and consistent view synthesis. We propose projective conditioning, which replaces raw camera parameters with a target-view projective cue that provides a stable 2D input. This reframes the task from a brittle geometric regression problem in ray space to a well-conditioned target-view image-to-image translation problem. Additionally, we introduce a masked autoencoding pretraining strategy tailored to this cue, enabling the use of large-scale uncalibrated data for pretraining. Our method shows improved fidelity and stronger cross-view consistency compared to ray-conditioned baselines on our view-consistency benchmark. It also achieves state-of-the-art quality on standard novel view synthesis benchmarks.

Figures (12)

• Figure 1: Projective conditioning enables robust novel view synthesis. We investigate what camera pose encoding best conditions a view synthesis model. Compared to the commonly used absolute valued Plücker ray conditioning by prior works jin_lvsmlarge_2025jiang_rayzerselfsupervised_2025, our proposed projection conditioning encodes scene-camera configuration as a relative transformation. Under various geometric transformations, this shows better robustness while the absolute conditioning signal fails due to the non-smoothness of transformations in the Plücker ray space.
• Figure 2: An overview of our proposed two-stage training pipeline. 1. Pretraining: This stage is self-supervised with the model conditioned on a set of context views and a randomly masked version of the target view itself (Masked Image). Its objective is to reconstruct the complete, original Ground Truth (GT) Target View. 2. Fine-Tuning: The context views are first unprojected into a unified 3D point cloud with extracted depth from perception models keetha_mapanythinguniversal_2025, which is then rasterized from the perspective of the target camera's frustum to create a point cloud projection image that provides geometric cues. The model is then fine-tuned to generate the final target image.
• Figure 3: Under a random global $\mathrm{SE}(3)$ transformation to the global coordinate system, ray-conditioned models jin_lvsmlarge_2025 produce degenerate results while projective conditioning remain robust.
• Figure 4: Qualitative Results on our Consistency Benchmark. Our method produces more geometrically consistent results. LVSM struggles to maintain geometric consistency, while RayZer and AnySplat fail to retrieve accurate camera parameters.
• Figure 5: Our training stages.Pretraining (top): The model reconstructs a target view from a randomly masked version of itself, conditioned on context views, using uncalibrated image data. Fine-tuning (bottom): The model is then fine-tuned to reconstruct the target view from a point-cloud projection image obtained by warping the context views into the target camera frustum.