VistaDepth: Improving far-range Depth Estimation with Spectral Modulation and Adaptive Reweighting

TL;DR

VistaDepth tackles the challenge of far-range monocular depth estimation by addressing two core issues: spectral bias in the backbone and the long-tail distribution of depth data. It introduces Latent Frequency Modulation (LFM) to refine latent features in the frequency domain and BiasMap to adaptively reweight the diffusion supervision across timesteps, all within a frozen Stable Diffusion backbone. The approach achieves state-of-the-art results among diffusion-based MDE methods, with substantial improvements in distant structures and data efficiency (e.g., competitive indoor and outdoor benchmarks using far less labeled data). These contributions advance robust, high-fidelity depth perception for applications like autonomous driving and AR, and open avenues for faster diffusion-based depth estimation through distillation and data-centric training enhancements.

Abstract

Monocular depth estimation (MDE) aims to infer per-pixel depth from a single RGB image. While diffusion models have advanced MDE with impressive generalization, they often exhibit limitations in accurately reconstructing far-range regions. This difficulty arises from two key challenges. First, the implicit multi-scale processing in standard spatial-domain models can be insufficient for preserving the fine-grained, high-frequency details crucial for distant structures. Second, the intrinsic long-tail distribution of depth data imposes a strong training bias towards more prevalent near-range regions. To address these, we propose VistaDepth, a novel diffusion framework designed for balanced and accurate depth perception. We introduce two key innovations. First, the Latent Frequency Modulation (LFM) module enhances the model's ability to represent high-frequency details. It operates by having a lightweight network predict a dynamic, content-aware spectral filter to refine latent features, thereby improving the reconstruction of distant structures. Second, our BiasMap mechanism introduces an adaptive reweighting of the diffusion loss strategically scaled across diffusion timesteps. It further aligns the supervision with the progressive denoising process, establishing a more consistent learning signal. As a result, it mitigates data bias without sacrificing training stability. Experiments show that VistaDepth achieves state-of-the-art performance for diffusion-based MDE, particularly excelling in reconstructing detailed and accurate depth in far-range regions.

Figures (5)

• Figure 1: (a) Illustration of the long-tail depth distribution in a typical outdoor scene. Near-range pixels are dense, while far-range pixels are sparse. (b) VistaDepth demonstrates superior performance across all depth ranges, with a disproportionately large gain in the challenging, data-sparse far-range regions, validating its effectiveness in mitigating data bias.
• Figure 2: VistaDepth Training Pipeline. An RGB image $\mathbf{x}$ and its depth map $\mathbf{d}$ are encoded into latents $\mathbf{z}^{x}$ and $\mathbf{z}_{0}^{d}$. Noise $\boldsymbol{\epsilon}_t$ is added to the depth latent for a random timestep $t$. The LFM-enhanced U-Net takes the concatenated pair $(\mathbf{z}^{x}, \mathbf{z}_{t}^{d})$ as input and is trained to predict the added noise. The LFM module, embedded in the decoder, refines features with content-aware frequency modulation (see Sec. \ref{['sec:Latent-Frequency-Modulation']}).
• Figure 3: VistaDepth Inference Pipeline. An input RGB image $\mathbf{x}$ is encoded into a conditioning latent $\mathbf{z}^x$. Separately, a tensor of pure Gaussian noise $\mathbf{z}_T^{d}$ is iteratively denoised by the LFM-enhanced U-Net. At each step, the network predicts the noise from the current latent $\mathbf{z}_t^{d}$, conditioned on $\mathbf{z}^x$. The final clean latent $\hat{\mathbf{z}}_0^{d}$ is passed through the VAE decoder to reconstruct the high-resolution depth map.
• Figure 4: Qualitative comparison of MDE across different datasets. VistaDepth demonstrates superior performance in capturing fine details at a distance (e.g., edges of doorways in NYU, distant cars in KITTI, windows in ETH3D and DIODE) and maintaining scene consistency (e.g., complex interior scene in ScanNet).
• Figure 5: Ablation on the temporal modulation $\gamma$. Training loss curves for different values of the ramp factor exponent from Sec. \ref{['sec:biasmap']}.