MAMo: Leveraging Memory and Attention for Monocular Video Depth Estimation

TL;DR

MAMo addresses the underutilization of temporal information in monocular depth estimation by introducing a memory-augmented, attention-based framework that can retrofit any monocular depth network into a video depth estimator. It maintains a memory of visual and displacement tokens across past frames, updates them to capture motion-equivariant features, and fuses them with current frame features through self- and cross-attention before decoding depth. The approach yields state-of-the-art or competitive results on KITTI, DDAD, and NYU-Depth V2 video datasets, while offering lower latency than many cost-volume-based video-depth models. This memory-attention paradigm enables robust, temporally-consistent depth predictions suitable for real-time or near-real-time applications in autonomous driving and robotics.

Abstract

We propose MAMo, a novel memory and attention frame-work for monocular video depth estimation. MAMo can augment and improve any single-image depth estimation networks into video depth estimation models, enabling them to take advantage of the temporal information to predict more accurate depth. In MAMo, we augment model with memory which aids the depth prediction as the model streams through the video. Specifically, the memory stores learned visual and displacement tokens of the previous time instances. This allows the depth network to cross-reference relevant features from the past when predicting depth on the current frame. We introduce a novel scheme to continuously update the memory, optimizing it to keep tokens that correspond with both the past and the present visual information. We adopt attention-based approach to process memory features where we first learn the spatio-temporal relation among the resultant visual and displacement memory tokens using self-attention module. Further, the output features of self-attention are aggregated with the current visual features through cross-attention. The cross-attended features are finally given to a decoder to predict depth on the current frame. Through extensive experiments on several benchmarks, including KITTI, NYU-Depth V2, and DDAD, we show that MAMo consistently improves monocular depth estimation networks and sets new state-of-the-art (SOTA) accuracy. Notably, our MAMo video depth estimation provides higher accuracy with lower latency, when omparing to SOTA cost-volume-based video depth models.

Figures (10)

• Figure 1: Our proposed MAMo (bottom) enables video depth estimation efficiently in a streaming fashion, by leveraging memory and attention. Monocular depth estimation fails to leverage temporal information (top), while existing cost-volume-based video depth models are computationally expensive (middle). For instance, for each inference, they require multiple image warping operations as well as significant memory usage and heavy computation to construct the cost volume(s).
• Figure 2: Overview of proposed MAMo method.
• Figure 3: Overview of proposed memory update scheme. To concisely illustrate the main idea of memory update, we omit some operations in the figure, e.g., self-attention on memory tokens (c.f. Section \ref{['sec:memory_attention']}), decoder feature carry-over (c.f. Section \ref{['sec:additional']}).
• Figure 4: Qualitative results on KITTI. We highlight (white boxes) regions where MAMo significantly improves depth estimation quality.
• Figure 5: RMSE vs. Latency on KITTI Eigen test set.