WildRayZer: Self-supervised Large View Synthesis in Dynamic Environments

TL;DR

WildRayZer tackles dynamic large-view synthesis by disentangling transient motion from static structure in a fully self-supervised, pose-free framework. It extends RayZer with a learned motion estimator and a masked scene encoder, using an analysis-by-synthesis workflow where residuals from a static renderer reveal dynamic regions and supervise pseudo-motion masks via a distillation process. The model is trained on Dynamic RealEstate10K (D-RE10K, 15K sequences) and D-RE10K-iPhone, and it consistently outperforms optimization-based and feed-forward baselines in both transient-region removal and full-frame NVS under sparse views. The approach enables scalable dynamic NVS without 3D supervision or ground-truth masks, and it introduces large-scale dynamic data and paired benchmarks to support transient-aware NVS research. The core objective is to learn a static-background renderer that is robust to dynamic content, achieved through masked-token rendering and copy–paste augmentation, with a final loss of the form $ \,L = \frac{1}{|\mathcal{I}_{\mathcal{B}}|} \sum_{\hat{I} \in \hat{\mathcal{I}}_{\mathcal{B}}} \left( \mathrm{MSE}(I,\hat{I}) + \lambda \, \mathrm{Percep}(I,\hat{I}) ight) $ guiding training.

Abstract

We present WildRayZer, a self-supervised framework for novel view synthesis (NVS) in dynamic environments where both the camera and objects move. Dynamic content breaks the multi-view consistency that static NVS models rely on, leading to ghosting, hallucinated geometry, and unstable pose estimation. WildRayZer addresses this by performing an analysis-by-synthesis test: a camera-only static renderer explains rigid structure, and its residuals reveal transient regions. From these residuals, we construct pseudo motion masks, distill a motion estimator, and use it to mask input tokens and gate loss gradients so supervision focuses on cross-view background completion. To enable large-scale training and evaluation, we curate Dynamic RealEstate10K (D-RE10K), a real-world dataset of 15K casually captured dynamic sequences, and D-RE10K-iPhone, a paired transient and clean benchmark for sparse-view transient-aware NVS. Experiments show that WildRayZer consistently outperforms optimization-based and feed-forward baselines in both transient-region removal and full-frame NVS quality with a single feed-forward pass.

Figures (9)

• Figure 1: Our self-supervised WildRayZer learns to render static novel views from dynamic images without any 3D or GT mask supervision. It extends the state-of-the-art self-supervised large view synthesis model RayZer to dynamic environments by adding a learned motion mask estimator and a masked 3D scene encoder.
• Figure 2: WildRayZer self-supervised learning framework.(a) Training. WildRayZer takes unposed, uncalibrated multi-view dynamic images $\mathcal{I}$ and predicts per-view camera parameters (intrinsics and relative poses), which are converted into pixel-aligned Plücker ray maps $\mathcal{R}$. A camera-only static renderer explains the rigid background; residuals between renderings $\hat{\mathcal{I}}_B$ and targets $\mathcal{I}_B$ highlight dynamic regions, which are sharpened by our pseudo-motion mask constructor (see \ref{['subsec:psuedo']} and \ref{['fig:pseudomask']}). We distill a motion estimator from these pseudo-masks and use it to gate dynamic image tokens before scene encoding; the same pseudo-masks also gate dynamic pixels in the photometric rendering loss. (b) Inference. Given dynamic input views $\mathcal{I}$, the model predicts camera parameters, motion masks, and a static scene representation in a single feed-forward pass. The motion estimator operates on the input views to mask dynamic tokens, and the renderer synthesizes transient-free novel views given the inferred scene representation and a target camera.
• Figure 3: Pseudo Motion Mask Pipeline. We fuse SSIM- and DINO-based dissimilarity into a saliency map, cluster DINO patch features to vote for dynamic patches, then refine the coarse patch mask to pixel resolution via morphological smoothing, small-component removal, and GrabCut rother2004grabcut.
• Figure 4: Qualitative Comparisons. Qualitative results on DRE10K-Mask (top two rows) and DRE10K-iPhone (bottom row). Compared to baselines, our method (1) more cleanly removes transient objects, (2) better handles cross-view completion (compare with RayZer + SAV baseline), and (3) better preserves global scene geometry (e.g., kitchens) and fine details (e.g., plants). SLS denotes Splotless-Splats sabour2025spotlesssplats.
• Figure 5: Qualitative Results. (1) First row: D-RE10K (no ground-truth novel views). (2) Second row: D-RE10K-iPhone. (3) Third and fourth rows: additional NVS results on DAVIS perazzi2016benchmark, where ground truth is also unavailable, demonstrating that WildRayZer generalizes to outdoor scenes and can mask unseen transient objects.