UMAMI: Unifying Masked Autoregressive Models and Deterministic Rendering for View Synthesis

TL;DR

UMAMI tackles the challenge of novel view synthesis from sparse inputs by unifying deterministic rendering with diffusion-based completion. It employs a bidirectional transformer to encode multi-view content and Plücker ray embeddings into a shared latent, feeding two heads: a fast deterministic renderer for well-constrained regions and a masked autoregressive diffusion head for unseen areas. The model is trained end-to-end with a combination of photometric, confidence, and diffusion losses, and uses a confidence-based sampling scheme that deterministically fills high-confidence tokens before diffusion-based completion of the rest. Across RealEstate10K and DL3DV, UMAMI achieves state-of-the-art or competitive results with significantly faster rendering than fully generative baselines, and demonstrates robust performance across interpolation, extrapolation, and varying input-view configurations.

Abstract

Novel view synthesis (NVS) seeks to render photorealistic, 3D-consistent images of a scene from unseen camera poses given only a sparse set of posed views. Existing deterministic networks render observed regions quickly but blur unobserved areas, whereas stochastic diffusion-based methods hallucinate plausible content yet incur heavy training- and inference-time costs. In this paper, we propose a hybrid framework that unifies the strengths of both paradigms. A bidirectional transformer encodes multi-view image tokens and Plucker-ray embeddings, producing a shared latent representation. Two lightweight heads then act on this representation: (i) a feed-forward regression head that renders pixels where geometry is well constrained, and (ii) a masked autoregressive diffusion head that completes occluded or unseen regions. The entire model is trained end-to-end with joint photometric and diffusion losses, without handcrafted 3D inductive biases, enabling scalability across diverse scenes. Experiments demonstrate that our method attains state-of-the-art image quality while reducing rendering time by an order of magnitude compared with fully generative baselines.

Figures (7)

• Figure 1: UMAMI synthesizes photorealistic novel views from sparse inputs. Shown are single-view generation, three-view extrapolation, and six-view reconstruction. The hybrid model fuses deterministic rendering with diffusion-based completion for unseen regions, yielding fast and consistent results without explicit 3D priors.
• Figure 2: UMAMI synthesizes target images from their camera poses and context views (each paired with its Plücker pose). During training, we randomly mask the target image, replace masked areas with learnable tokens, and concatenate these with the target's Plücker embedding. Input views are also tokenized. A Transformer processes both tokenized inputs and the masked target representation to produce a latent $z$. This code inputs to two MLP heads: a deterministic head ($\varphi$) outputs RGB and confidence, while a diffusion head ($\phi$) models the distribution of target tokens conditioned on $\mathbf{z}$. The model is trained end-to-end using a weighted loss combination (Section \ref{['subsec:training-loss']}). At inference, the target image is initialized with learned masked tokens for our proposed hybrid sampling (Section \ref{['subsec:hybrid-sampler']}).
• Figure 3: Hybrid Masked Autoregressive Sampler. Top: Conventional Masked Generative samplers li2024autoregressivechang2022maskgitli2023mage predict multiple tokens simultaneously using random ordering. Bottom (Ours): A deterministic first pass for high confidence tokens, followed by simultaneous random-order sampling for the remaining tokens, significantly boosts rendering times for the NVS task.
• Figure 4: Qualitative results on Re10K Evaluation of UMAMI on the challenging Re10K-2View-Extra extrapolation set, comparing it with LVSM jin2024lvsmlargeviewsynthesis, MVSplat chen2024mvsplat, and SEVA zhou2025stable. UMAMI not only renders sharp details in observed regions but also generates plausible content for unseen areas. More results can be viewed in the Appendix.
• Figure 5: Failure cases. Our method may produce noticeable artifacts when target camera poses are too distant from the input view. Increasing the scale of training data and model parameters could improve the robustness of UMAMI.