METER: a mobile vision transformer architecture for monocular depth estimation

TL;DR

METER is proposed, a novel lightweight vision transformer architecture capable of achieving state of the art estimations and low latency inference performances on the considered embedded hardwares: NVIDIA Jetson TX1 andNVIDIA Jetson Nano.

Abstract

Depth estimation is a fundamental knowledge for autonomous systems that need to assess their own state and perceive the surrounding environment. Deep learning algorithms for depth estimation have gained significant interest in recent years, owing to the potential benefits of this methodology in overcoming the limitations of active depth sensing systems. Moreover, due to the low cost and size of monocular cameras, researchers have focused their attention on monocular depth estimation (MDE), which consists in estimating a dense depth map from a single RGB video frame. State of the art MDE models typically rely on vision transformers (ViT) architectures that are highly deep and complex, making them unsuitable for fast inference on devices with hardware constraints. Purposely, in this paper, we address the problem of exploiting ViT in MDE on embedded devices. Those systems are usually characterized by limited memory capabilities and low-power CPU/GPU. We propose METER, a novel lightweight vision transformer architecture capable of achieving state of the art estimations and low latency inference performances on the considered embedded hardwares: NVIDIA Jetson TX1 and NVIDIA Jetson Nano. We provide a solution consisting of three alternative configurations of METER, a novel loss function to balance pixel estimation and reconstruction of image details, and a new data augmentation strategy to improve the overall final predictions. The proposed method outperforms previous lightweight works over the two benchmark datasets: the indoor NYU Depth v2 and the outdoor KITTI.

Figures (7)

• Figure 1: METER depth map predictions (third-row) over the KITTI and NYU Depth v2 datasets. GT depth maps are resized to match METER output resolution. The depth maps are converted in RGB format with a perceptually uniform colormap (Plasma-reversed) extracted from the ground truth (second-row), for a better view.
• Figure 2: Overview of METER encoder-decoder network structure. The processing flow, i.e. the sequence of operations and the skip-connection, is represented with a blue dashed arrow. The (H, W, C) format refers to the input-output spatial dimensions, while the $\uparrow$ and $\downarrow$ refers to the feature resolution upsampling and downsampling.
• Figure 3: Illustration of an augmented sample with the proposed shifting strategy. The shifting factors ($\beta$, $\gamma$, $\eta$, and $S$) are set as their maximum and minimum values, i.e. $\{0.9, -10\}$ and $\{1.1, +10\}$ respectively. The min/max depth ranges for the regions of interest are given through the respective colored bars.
• Figure 4: A graphical comparison among METER (S, XS, XXS) configurations. For a better visualization, we apply to depth images and difference maps uniform colormaps with the same depth range. Precisely, in the ground truth (GT) and predicted depth maps (Pred) a lower color intensity corresponds to further distances, while in the difference map (Diff $= |$GT $-$ Pred$|$) a lower color intensity corresponds to a smaller error.
• Figure 5: Qualitative comparison of a predicted frame taking into account different loss components. For a better visualization, we apply to the depth images and to the difference maps uniform colormaps with the same depth range. Precisely, in the ground truth (GT) and predicted depth maps (Pred) a lower color intensity corresponds to further distances, while in the difference map (Diff $= |$GT $-$ Pred$|$) a lower color intensity corresponds to a smaller error.