ARSS: Taming Decoder-only Autoregressive Visual Generation for View Synthesis From Single View

TL;DR

<3-5 sentence high-level summary> ARSS tackles the problem of generating temporally coherent novel views from a single image along a predefined camera trajectory. It introduces a decoder-only autoregressive framework that uses a video tokenizer to encode multi-view sequences, a camera autoencoder to inject 3D positional guidance via Plücker raymaps, and a permutation-based token ordering to maintain temporal causality while leveraging spatial context. The method achieves competitive performance with diffusion-based NVS on RealEstate-10K, ACID, and DL3DV in zero-shot settings, and demonstrates strong long-horizon stability. This work highlights a path toward autoregressive, causally-structured view synthesis and suggests future improvements in tokenizers and high-resolution training.

Abstract

Diffusion models have achieved impressive results in world modeling tasks, including novel view generation from sparse inputs. However, most existing diffusion-based NVS methods generate target views jointly via an iterative denoising process, which makes it less straightforward to impose a strictly causal structure along a camera trajectory. In contrast, autoregressive (AR) models operate in a causal fashion, generating each token based on all previously generated tokens. In this work, we introduce ARSS, a novel framework that leverages a GPT-style decoder-only AR model to generate novel views from a single image, conditioned on a predefined camera trajectory. We employ an off-the-shelf video tokenizer to map continuous image sequences into discrete tokens and propose a camera encoder that converts camera trajectories into 3D positional guidance. Then to enhance generation quality while preserving the autoregressive structure, we propose an autoregressive transformer module that randomly permutes the spatial order of tokens while maintaining their temporal order. Qualitative and quantitative experiments on public datasets demonstrate that our method achieves overall performance comparable to state-of-the-art view synthesis approaches based on diffusion models. Project page: https://wbteng9526.github.io/arss/.

Figures (11)

• Figure 1: Illustration of ARSS. Given a source input image and camera trajectory, ARSS can generate photorealistic and 3D consistent novel views. Although a lot of previous methods tackle the same task with other generative model like diffusion models rombach2022highho2022video, ARSS is the first that leverages decoder-only causal transformer and generate multi-views with a next-token prediction style.
• Figure 2: Overall architecture of our proposed method. Left: we apply a video tokenizer to convert image sequence into latent codes. We also apply a camera autoencoder to map camera Pl√ºcker raymap to latent camera tokens. The camera tokens are inserted before visual tokens as a 3D positional instruction. Right: the interleaved sequence is the input of a decoder-only causal transformer. The tokens of the first view are the condition tokens thus always visible to all the subsequent tokens. We use the ground truth sequence from the tokenization process to supervise the weights of autoregression model.
• Figure 3: Qualitative Visualization. Qualitative comparison between ARSS with other diffusion-based and feed-forward transformer-based methods on ReaEstate10K and ACID datasets. Diffusion-based methods such as SEVA and Genwarp often suffer from distortions and inaccurate camera pose alignment, while the feed-forward transformer-based LVSM produces results that are noticeably blurry along boundaries. In contrast, ARSS generates geometrically consistent and sharp views across diverse scenes.
• Figure 4: Qualitative Visualization. Zero-shot novel view synthesis comparison between ARSS with other diffusion-based and feed-forward transformer-based methods on DL3DV benchmark ling2024dl3dv. MotionCtrl and Genwarp exhibit distortions due to incorrect camera pose alignment, while LVSM produces results that are noticeably blurry. Our proposed method, ARSS, generates sharp views with geometric consistency
• Figure 5: View Generation Results. Zero-shot novel view synthesis visualization on AI Generated betker2023improving images. The results demonstrate the strong generalizability of our method, generating consistent and high-fidelity novel views even when applied to out-of-distribution, synthetically generated inputs.