Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

True Self-Supervised Novel View Synthesis is Transferable

Thomas W. Mitchel, Hyunwoo Ryu, Vincent Sitzmann

TL;DR

This work reframes novel view synthesis (NVS) around transferability: the ability to render the same camera trajectory in a new scene using poses from another sequence. It introduces XFactor, a geometry-free, self-supervised model that achieves true NVS by coupling a stereo-monocular pose encoder with a renderer and a transferability-driven augmentation strategy, avoiding explicit 3D priors. A new metric, True Pose Similarity (TPS), quantifies how well pose latents transfer across scenes, and large-scale experiments on RE10K, DL3DV, MVImgNet, and CO3Dv2 show XFactor outperforms prior pose-free methods (RayZer and RUST) in transferability and pose fidelity. The results demonstrate that transferable latent poses can support robust NVS without geometric inductive biases, revealing a path toward geometry-free 3D reasoning in real-world video.

Abstract

In this paper, we identify that the key criterion for determining whether a model is truly capable of novel view synthesis (NVS) is transferability: Whether any pose representation extracted from one video sequence can be used to re-render the same camera trajectory in another. We analyze prior work on self-supervised NVS and find that their predicted poses do not transfer: The same set of poses lead to different camera trajectories in different 3D scenes. Here, we present XFactor, the first geometry-free self-supervised model capable of true NVS. XFactor combines pair-wise pose estimation with a simple augmentation scheme of the inputs and outputs that jointly enables disentangling camera pose from scene content and facilitates geometric reasoning. Remarkably, we show that XFactor achieves transferability with unconstrained latent pose variables, without any 3D inductive biases or concepts from multi-view geometry -- such as an explicit parameterization of poses as elements of SE(3). We introduce a new metric to quantify transferability, and through large-scale experiments, we demonstrate that XFactor significantly outperforms prior pose-free NVS transformers, and show that latent poses are highly correlated with real-world poses through probing experiments.

True Self-Supervised Novel View Synthesis is Transferable

TL;DR

This work reframes novel view synthesis (NVS) around transferability: the ability to render the same camera trajectory in a new scene using poses from another sequence. It introduces XFactor, a geometry-free, self-supervised model that achieves true NVS by coupling a stereo-monocular pose encoder with a renderer and a transferability-driven augmentation strategy, avoiding explicit 3D priors. A new metric, True Pose Similarity (TPS), quantifies how well pose latents transfer across scenes, and large-scale experiments on RE10K, DL3DV, MVImgNet, and CO3Dv2 show XFactor outperforms prior pose-free methods (RayZer and RUST) in transferability and pose fidelity. The results demonstrate that transferable latent poses can support robust NVS without geometric inductive biases, revealing a path toward geometry-free 3D reasoning in real-world video.

Abstract

In this paper, we identify that the key criterion for determining whether a model is truly capable of novel view synthesis (NVS) is transferability: Whether any pose representation extracted from one video sequence can be used to re-render the same camera trajectory in another. We analyze prior work on self-supervised NVS and find that their predicted poses do not transfer: The same set of poses lead to different camera trajectories in different 3D scenes. Here, we present XFactor, the first geometry-free self-supervised model capable of true NVS. XFactor combines pair-wise pose estimation with a simple augmentation scheme of the inputs and outputs that jointly enables disentangling camera pose from scene content and facilitates geometric reasoning. Remarkably, we show that XFactor achieves transferability with unconstrained latent pose variables, without any 3D inductive biases or concepts from multi-view geometry -- such as an explicit parameterization of poses as elements of SE(3). We introduce a new metric to quantify transferability, and through large-scale experiments, we demonstrate that XFactor significantly outperforms prior pose-free NVS transformers, and show that latent poses are highly correlated with real-world poses through probing experiments.
Paper Structure (17 sections, 12 equations, 2 figures, 2 tables)

This paper contains 17 sections, 12 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Oracle, Semi-Oracle, and Geometric Methods.
  4. Unsupervised Geometry-Free Latent Pose Methods.
  5. Method
  6. Novel View Synthesis as Latent Variable Modeling
  7. True Novel View Synthesis is Transferable
  8. Quantifying Transferability with True Pose Similarity (TPS)
  9. Solving Two Principal Problems: Interpolation and Information Leakage
  10. XFactor: A Model for True Self-Supervised Novel View Synthesis
  11. Experiments
  12. Transferability
  13. Pose Probe
  14. Ablations
  15. Discussion
  16. ...and 2 more sections

Figures (2)

  • Figure 1: XFactor combines a $[\textcolor{xpe}{PoseEnc}{}, \, \textcolor{xrender}{Render}{}]$stereo-monocular model with our proposed transferability objective to learn transferable camera pose latents. Given a pair of input images, we apply two augmentations that minimize pixel content overlap while preserving pose information, such as inverse masking. The stereo $\textcolor{xpe}{PoseEnc}$ extracts the relative pose latent from the first pair. Then, given the context image from the second pair and the first's target pose, the renderer is asked to reconstruct the second's target.
  • Figure :