StableDPT: Temporal Stable Monocular Video Depth Estimation

TL;DR

This paper tackles temporal instability in monocular video depth estimation by augmenting image based depth models with a global temporal cross attention head that leverages keyframes sampled across the entire video. StableDPT uses an off the shelf ViT encoder and a DPT style head with temporal layers that cross attend to distant frames, enabling robust depth predictions without post processing. A strided inference strategy anchors context with global keyframes to process arbitrary length videos efficiently while avoiding scale drift. The method is trained on a mix of image and video data with losses including SSI Trim, Gradient Matching, and Temporal Gradient Matching, achieving competitive accuracy and temporal stability with roughly 2x faster inference on standard hardware. The approach is lightweight, end to end trainable, and generalizable to other dense prediction tasks.

Abstract

Applying single image Monocular Depth Estimation (MDE) models to video sequences introduces significant temporal instability and flickering artifacts. We propose a novel approach that adapts any state-of-the-art image-based (depth) estimation model for video processing by integrating a new temporal module - trainable on a single GPU in a few days. Our architecture StableDPT builds upon an off-the-shelf Vision Transformer (ViT) encoder and enhances the Dense Prediction Transformer (DPT) head. The core of our contribution lies in the temporal layers within the head, which use an efficient cross-attention mechanism to integrate information from keyframes sampled across the entire video sequence. This allows the model to capture global context and inter-frame relationships leading to more accurate and temporally stable depth predictions. Furthermore, we propose a novel inference strategy for processing videos of arbitrary length avoiding the scale misalignment and redundant computations associated with overlapping windows used in other methods. Evaluations on multiple benchmark datasets demonstrate improved temporal consistency, competitive state-of-the-art performance and on top 2x faster processing in real-world scenarios.

Figures (8)

• Figure 1: Left: Our model, StableDPT, can leverage information from the entire video sequence to produce temporally stable and accurate depth maps from monocular video input with complex camera and scene dynamics. Right: our keyframes can effectively capture global context of the video, and anchor current predictions via temporal module with cross-attention.
• Figure 2: Overall architecture of our model. Left: The encoder extracts features from each frame independently, while the head processes these features with global context temporal layers that integrate information from keyframes sampled from the entire video sequence. The output is a depth map for each frame in the input video. Right: Detailed view of the temporal layer, which uses cross-attention mechanisms to align features with information from the global context.
• Figure 3: Visualization of attention maps from the encoder and decoder attention layers. Top row: Attention maps from the encoder show spatial attention within individual frames, capturing relationships between different regions of the same image. Bottom row: Attention maps from the decoder temporal layers illustrate attention across time, highlighting how the temporal layer informs current features about surrounding frames, past and future.
• Figure 4: Illustration of different inference strategies for processing videos of arbitrary length $N$. (a) Batched inference divides the video into non-overlapping batches of fixed length, leading to scale misalignment between batches. (b) VDA inference processes overlapping windows and interpolates the overlapping regions, which requires processing more frames. (c) Strided inference samples frames at regular intervals from the entire video, ensuring each frame is processed with context from the whole video.
• Figure 5: Training compute comparison in GPU hours. Our method requires significantly less compute compared to other methods. Green/orange indicates transformer/diffusion based methods. Size indicates inference time.