Toward A Better Understanding of Monocular Depth Evaluation

TL;DR

This paper tackles inconsistent monocular depth evaluation by introducing a sensitivity-based framework that measures metric responses to perturbations of ground-truth depth. It enumerates six perturbations and uses an exchange-rate $R(A;B|P)=\frac{A'(0)}{B'(0)}$ to compare metrics, augmented by synthetic data and human judgments via new visualization tools. The authors propose Sensitivity Aligned Composition (SAC) and introduce RelNormal as a curvature-sensitive base metric, yielding SAWA-H as a human-aligned composite; they show that existing metrics under-respond to curvature and misalign with human perception. The work provides practical tools, datasets, and code to enable principled, application-driven metric design for monocular depth evaluation, with broad implications for evaluating and developing MDE methods.

Abstract

Monocular depth estimation is an important task with rapid progress, but how to evaluate it is not fully resolved, as evidenced by a lack of standardization in existing literature and a large selection of evaluation metrics whose trade-offs and behaviors are not fully understood. This paper contributes a novel, quantitative analysis of existing metrics in terms of their sensitivity to various types of perturbations of ground truth, emphasizing comparison to human judgment. Our analysis reveals that existing metrics are severely under-sensitive to curvature perturbation such as making smooth surfaces bumpy. To remedy this, we introduce a new metric based on relative surface normals, along with new depth visualization tools and a principled method to create composite metrics with better human alignment. Code and data are available at: https://github.com/princeton-vl/evalmde.

Figures (21)

• Figure 1: Left: Surface orientation perturbation (side view). Orientation of prediction (red) equals rotating that of ground truth (green) around x-axis by $-10^\circ$. In side views through the paper, we show simple geometry whose depth is the same on each row. Middle & Right: Camera intrinsics perturbation. Middle (side view of 3D shape): Prediction is made under wrong focal length (2x ground truth focal length), while predicted geometry (red) has shape similar to ground truth (green). Predicted geometry can overlap well with ground truth through translation (red dashed). Right (depth value of each row): Predicted depth substantially differs from ground truth depth.
• Figure 2: Exchange rate between each metric and AbsRel with no alignment. Because absolute scale of the exchange rate is not important, we normalize each row for better visualization. Perturbations from left to right: surface orientation, camera intrinsics, relative scale, curvature (high frequency), curvature (low frequency), affine transform of depth, affine transform of disparity, and boundary.
• Figure 3: Top: input image and visualization of a depth using existing tools. Defects like a wavy wall are not apparent due to interference of texture. Bottom: Textureless Relighting and Projected Contours make geometric defects more apparent. Bumps on the wall are obvious.
• Figure 4: Sensitivity: Each row in red is a vector of exchange rates between a metric and human judgment, reflecting the sensitivity of a metric to different perturbations, using human judgment as a reference. Within a vector, a higher value means higher sensitivity of the metric to a perturbation, relative to humans. Human judgment (last row) against itself results in a vector of all 1s. Each vector can be arbitrarily scaled, equivalent to changing the unit of the metric; here the unit of the metric is rescaled such that the L2 norm of the vector is the same as a vector of all 1s, to facilitate comparisons across rows. All rows except the last four are existing metrics. Similarity to human sensitivity: cosine similarity between each sensitivity vector and the human sensitivity vector (all 1s). SAWA-H Weights: SAWA-H is a new, composite metric that is a weighted average of base metrics, with weights optimized to align with human sensitivity. RelNormal is a new base metric we introduce to allow better human alignment. The first column lists weights for averaging metrics under rescaled units, reflecting relative contributions invariant to the original choices of metric units; the second column lists equivalent weights for averaging metrics under their original units.
• Figure 5: Performance of state-of-the-art monocular depth estimation methods under SAWA-H, averaged over 9 datasets. For component metrics of SAWA-H, the weight used to combine them to obtain SAWA-H under original metric units are listed next to metric names. Align w/ GT Depth, Un-proj. w/ GT Cam. Intr. refers to methods whose output is first aligned with ground truth depth then un-projected using ground truth camera intrinsics; No Align., Un-proj. w/ GT Cam. Intr. refers to methods whose output is directly un-projected using ground truth camera intrinsics; No Align., Un-proj. w/ Pred. Cam. Intr. refers to methods whose output is directly un-projected using predicted camera intrinsics. Note that results are not directly comparable if one method is aligned with ground truth or un-projected using ground truth camera intrinsics while the other is not.