Pixel-Perfect Visual Geometry Estimation

TL;DR

Pixel-Perfect Visual Geometry Estimation tackles flying pixels and edge artifacts in monocular and video depth estimation by performing diffusion directly in the pixel space. The authors introduce Pixel-Perfect Depth (PPD) and Pixel-Perfect Video Depth (PPVD), powered by Semantics-Prompted Diffusion Transformers (SP-DiT) and Semantics-Consistent DiT (SC-DiT), plus a Cascaded DiT (Cas-DiT) architecture and Reference-Guided Token Propagation (RGTP) for efficiency and temporal coherence. The approach eliminates VAE-induced edge artifacts and achieves state-of-the-art results among generative monocular/video depth models, with an edge-aware point cloud evaluation demonstrating cleaner geometry. This work enables high-quality, flying-pixel-free depth maps suitable for robotics and augmented reality, and provides mechanisms for robust temporal consistency in videos.

Abstract

Recovering clean and accurate geometry from images is essential for robotics and augmented reality. However, existing geometry foundation models still suffer severely from flying pixels and the loss of fine details. In this paper, we present pixel-perfect visual geometry models that can predict high-quality, flying-pixel-free point clouds by leveraging generative modeling in the pixel space. We first introduce Pixel-Perfect Depth (PPD), a monocular depth foundation model built upon pixel-space diffusion transformers (DiT). To address the high computational complexity associated with pixel-space diffusion, we propose two key designs: 1) Semantics-Prompted DiT, which incorporates semantic representations from vision foundation models to prompt the diffusion process, preserving global semantics while enhancing fine-grained visual details; and 2) Cascade DiT architecture that progressively increases the number of image tokens, improving both efficiency and accuracy. To further extend PPD to video (PPVD), we introduce a new Semantics-Consistent DiT, which extracts temporally consistent semantics from a multi-view geometry foundation model. We then perform reference-guided token propagation within the DiT to maintain temporal coherence with minimal computational and memory overhead. Our models achieve the best performance among all generative monocular and video depth estimation models and produce significantly cleaner point clouds than all other models.

Figures (8)

• Figure 1: Visual comparison with existing depth foundation models. Discriminative models such as Depth Anything v2 and generative models such as Marigold, due to their inherent modeling paradigms or architectural limitations, produce substantial flying pixels. In contrast, our model estimates depth maps that produce high-quality, flying-pixel-free point clouds without any additional refinement or post-processing.
• Figure 2: Pixel diffusion vs. latent diffusion. GT(VAE reconstruction) denotes the ground truth depth map after VAE reconstruction. Existing generative models ke2024marigold use a VAE to compress inputs into the latent space, inevitably introducing flying pixels at edges and details. In contrast, our model directly performs diffusion in pixel space, avoiding these issues. Depth maps are visualized on the point clouds.
• Figure 3: Overview of Pixel-Perfect Depth. Given an input image concatenated with noise, we feed it into the proposed Cascade DiT. The image is also processed by a pretrained encoder from Vision Foundation Models to extract high-level semantics, forming our Semantics-Prompted DiT. We perform diffusion generation directly in pixel space without using any VAE.
• Figure 4: Overview of Pixel-Perfect Video Depth. Given a sequence of video frames concatenated with noise, we feed it into the proposed Cascade DiT. The video is also processed by a multi-view geometry-based model to capture spatiotemporally consistent semantics, forming our Semantics-Consistent DiT. In the subsequent DiT, to ensure temporal coherence within the single-view transformer, we introduce a reference-guided token propagation strategy, where sparse reference tokens propagate scale and shift information across frames.
• Figure 5: Comparison with existing depth foundation models. Our PPD preserves more fine-grained details than Depth Anything v2 yang2024depthv2 and MoGe 2 moge2, while demonstrating significantly higher robustness compared to Depth Pro depthpro.