Recollection from Pensieve: Novel View Synthesis via Learning from Uncalibrated Videos

TL;DR

This work tackles novel view synthesis from uncalibrated videos by introducing a two-stage self-supervised framework. Stage 1 performs implicit scene reconstruction to learn robust latent correspondences without explicit 3D priors, while Stage 2 enforces 3D consistency by predicting explicit Gaussian primitives and applying Gaussian Splatting and depth reprojection losses. An interpolation-based inference strategy mitigates challenges from two-view inputs, and the approach achieves strong NVS quality and camera pose estimation without calibration guidance. The method enables scalable training on diverse, uncalibrated video data with practical implications for 3D vision in unconstrained settings.

Abstract

Currently almost all state-of-the-art novel view synthesis and reconstruction models rely on calibrated cameras or additional geometric priors for training. These prerequisites significantly limit their applicability to massive uncalibrated data. To alleviate this requirement and unlock the potential for self-supervised training on large-scale uncalibrated videos, we propose a novel two-stage strategy to train a view synthesis model from only raw video frames or multi-view images, without providing camera parameters or other priors. In the first stage, we learn to reconstruct the scene implicitly in a latent space without relying on any explicit 3D representation. Specifically, we predict per-frame latent camera and scene context features, and employ a view synthesis model as a proxy for explicit rendering. This pretraining stage substantially reduces the optimization complexity and encourages the network to learn the underlying 3D consistency in a self-supervised manner. The learned latent camera and implicit scene representation have a large gap compared with the real 3D world. To reduce this gap, we introduce the second stage training by explicitly predicting 3D Gaussian primitives. We additionally apply explicit Gaussian Splatting rendering loss and depth projection loss to align the learned latent representations with physically grounded 3D geometry. In this way, Stage 1 provides a strong initialization and Stage 2 enforces 3D consistency - the two stages are complementary and mutually beneficial. Extensive experiments demonstrate the effectiveness of our approach, achieving high-quality novel view synthesis and accurate camera pose estimation, compared to methods that employ supervision with calibration, pose, or depth information. The code is available at https://github.com/Dwawayu/Pensieve.

Figures (4)

• Figure 1: Pipeline. Given an $N$-frame uncalibrated video (with $N = 4$ shown), our method first extracts per-frame features $\mathbf{F}^s$ using a shared-weight Per-Frame Transformer. Features from all frames are passed through a Camera Transformer to predict per-frame camera parameters $\mathbf{K}$ and $\mathbf{P}$. A sampled proper subset of features (e.g., $\{ \mathbf{F}^s_2, \mathbf{F}^s_4 \}$) is fed into the Context Transformer to predict the corresponding context features $\mathbf{F}^c$ and pixel-aligned Gaussian primitives $\mathbf{G}^c$ in the camera coordinate system. Subsequently, we rerender each frame and compute the rendering loss, as illustrated by the example of the first frame. For LVSM rendering, we apply Plücker coordinates embedding to represent camera features, and use the View Synthesis Transformer to predict the target image $\hat{\mathbf{I}}^M_1$. In Stage 2, we warp the Gaussian primitives into world coordinates, and concatenate them to additionally render the target image $\hat{\mathbf{I}}^G_1$ explicitly using the predicted camera parameters.
• Figure 2: Novel view synthesis comparisons on RealEstate10K re10k. Our method better aligns with ground-truth poses and image content.
• Figure 3: Comparison of the rendered depth maps with and without Stage 1 pretraining.
• Figure 4: Comparison of the camera trajectories with and without Stage 2 alignment.