RayZer: A Self-supervised Large View Synthesis Model

TL;DR

RayZer demonstrates that self-supervised training from unposed, uncalibrated multi-view images can yield a scalable, transformer-based 3D vision model with strong 3D awareness. By predicting camera parameters, converting to Plücker ray maps, and using a latent set scene representation with a learned rendering decoder, it achieves competitive novel view synthesis without any 3D supervision. Evaluations across DL3DV RealEstate and Objaverse show RayZer matching or exceeding oracle methods in many cases, and ablations highlight the importance of Plücker rays and pose-first design. The approach highlights the potential to reduce reliance on ground-truth camera poses and 3D geometry in future large-scale 3D vision systems.

Abstract

We present RayZer, a self-supervised multi-view 3D Vision model trained without any 3D supervision, i.e., camera poses and scene geometry, while exhibiting emerging 3D awareness. Concretely, RayZer takes unposed and uncalibrated images as input, recovers camera parameters, reconstructs a scene representation, and synthesizes novel views. During training, RayZer relies solely on its self-predicted camera poses to render target views, eliminating the need for any ground-truth camera annotations and allowing RayZer to be trained with 2D image supervision. The emerging 3D awareness of RayZer is attributed to two key factors. First, we design a self-supervised framework, which achieves 3D-aware auto-encoding of input images by disentangling camera and scene representations. Second, we design a transformer-based model in which the only 3D prior is the ray structure, connecting camera, pixel, and scene simultaneously. RayZer demonstrates comparable or even superior novel view synthesis performance than ``oracle'' methods that rely on pose annotations in both training and testing. Project: https://hwjiang1510.github.io/RayZer/

Figures (10)

• Figure 1: Our proposed self-supervised training framework. This is an abstract design that we later operationalize with our RayZer model (illustrated in Fig. \ref{['fig: model']} and Sec. \ref{['sec: method']}). We divide the input images into two sets $\mathcal{I}_\mathcal{A}$ and $\mathcal{I}_\mathcal{B}$. We predict the scene representation from $\mathcal{I}_\mathcal{A}$, and use the predicted cameras of $\mathcal{I}_\mathcal{B}$ (shown in orange) to render the scene. We leverage photometric loss between raw input $\mathcal{I}_\mathcal{B}$ and its prediction $\hat{\mathcal{I}}_\mathcal{B}$ for training.
• Figure 2: RayZer self-supervised learning framework. RayZer takes in unposed and uncalibrated multi-view images $\mathcal{I}$ and predicts per-view camera parameters and a scene representation, which supports novel view rendering. (Left) RayZer first estimates camera parameters, where one view is selected as the canonical reference view (in blue box). RayZer predicts the intrinsics and the relative camera poses $\mathcal{P}$ of all views. The predicted cameras are then converted into pixel-aligned Pl端cker ray maps $\mathcal{R}$. (Middle) RayZer uses a subset of input images, $\mathcal{I}_\mathcal{A}$, as well as their previously predicted camera Pl端cker ray maps, $\mathcal{R}_\mathcal{A}$, to predict a latent scene representation. Here, the Pl端cker ray maps, $\mathcal{R}_\mathcal{A}$, serve as an effective condition for scene reconstruction. (Right) RayZer can render a target image given the scene representation $\mathbf{z}^{*}$ and a target camera. During training, we use $\mathcal{R}_\mathcal{B}$, which is the previously predicted cameras Pl端cker ray maps of $\mathcal{I}_\mathcal{B}$, to render $\hat{\mathcal{I}}_\mathcal{B}$. This allows training RayZer end-to-end with self-supervised photometric losses between inputs $\mathcal{I}_\mathcal{B}$ and their renderings $\hat{\mathcal{I}}_\mathcal{B}$.
• Figure 3: Visualization results on RealEstate and DL3DV. We compare RayZer with "oracle" methods GS-LRM and LVSM, which use COLMAP pose annotations in both training and testing. Our self-supervised RayZer model does not use any pose annotations. Generally, RayZer performs on par with "oracle" methods (first row), and can outperform them on cases that COLMAP usually struggles to handle, e.g., glasses and white walls (highlighted with red boxes). The results verify our analysis on the problems of using COLMAP in Sec. \ref{['sec: main_results']}.
• Figure 4: Visualization results on Objaverse. RayZer performs on par with LVSM and outperforms the supervised method PF-LRM.
• Figure 5: Visualization of RayZer predicted cameras learned with self-supervision. We visualize 3 out of 5 rendered views due to space limit, where the image index is highlighted by its color.