Visual Autoregressive Modelling for Monocular Depth Estimation

TL;DR

This paper tackles monocular depth estimation by leveraging visual autoregressive priors (VAR) as an alternative to diffusion-based priors. It adapts a pre-trained VAR model (Switti) to depth by introducing scale-wise conditional upsampling, a re-encoding strategy, and a classifier-free guidance-based sampling scheme across $10$ autoregressive stages, finetuned with a modest synthetic dataset of $p_{ ext{data}}$-driven samples. The approach achieves state-of-the-art indoor performance under constrained data (finetuning on $74{,}000$ synthetic samples) and competitive outdoor generalization on KITTI, ETH3D, and DIODE, demonstrating the viability of autoregressive priors for geometry-aware depth estimation. Overall, the work positions autoregressive priors as a data-efficient, geometry-aware complement to diffusion-based methods, with practical implications for 3D vision tasks and robotics.

Abstract

We propose a monocular depth estimation method based on visual autoregressive (VAR) priors, offering an alternative to diffusion-based approaches. Our method adapts a large-scale text-to-image VAR model and introduces a scale-wise conditional upsampling mechanism with classifier-free guidance. Our approach performs inference in ten fixed autoregressive stages, requiring only 74K synthetic samples for fine-tuning, and achieves competitive results. We report state-of-the-art performance in indoor benchmarks under constrained training conditions, and strong performance when applied to outdoor datasets. This work establishes autoregressive priors as a complementary family of geometry-aware generative models for depth estimation, highlighting advantages in data scalability, and adaptability to 3D vision tasks. Code available at "https://github.com/AmirMaEl/VAR-Depth".

Figures (3)

• Figure 1: Overview of the proposed fine-tuning protocol. During model training the RGB image $\mathbf{c}_k$ is reprojected using a MLP to align the dimension $\mathbf{c}_k$ with the text dimension $\mathbf{c}_L$. The modified input each scale $k$ to the VAR transformer then consists of both the ground truth depth token $\mathbf{r}_k$ and the conditional encoding $\mathbf{c}_L$.
• Figure 2: Overview of proposed sampling procedure. During sampling the prediction from the previous scale $\mathbf{\hat{r}}_{k-1}$ and the conditional reprojected encodings $\mathbf{c}_L$ are processed by the VAR transformer. After this processing step the VAR prediction $\mathbf{\hat{r}}^u_k=\mathbf{\hat{f}}_k$ is fed into the conditional upsampling network $U(\cdot)$, along with the condition encoding $\mathbf{c}_L$ to predict the output $\mathbf{\hat{r}}^c_k$. Finally both components are combined using classifier-free guidance to form the upsampled depth prediction $\mathbf{\hat{r}}_k$.
• Figure 3: Predictions of Indoor and Outdoor experiments. Additionally we include the Ground truth and the input image provided by the test sets.