InSpaceType: Dataset and Benchmark for Reconsidering Cross-Space Type Performance in Indoor Monocular Depth

TL;DR

This work marks the first in-depth investigation of performance variances across space types and, more importantly, releases useful tools, including datasets and codes, to closely examine your pretrained depth models.

Abstract

Indoor monocular depth estimation helps home automation, including robot navigation or AR/VR for surrounding perception. Most previous methods primarily experiment with the NYUv2 Dataset and concentrate on the overall performance in their evaluation. However, their robustness and generalization to diversely unseen types or categories for indoor spaces (spaces types) have yet to be discovered. Researchers may empirically find degraded performance in a released pretrained model on custom data or less-frequent types. This paper studies the common but easily overlooked factor-space type and realizes a model's performance variances across spaces. We present InSpaceType Dataset, a high-quality RGBD dataset for general indoor scenes, and benchmark 13 recent state-of-the-art methods on InSpaceType. Our examination shows that most of them suffer from performance imbalance between head and tailed types, and some top methods are even more severe. The work reveals and analyzes underlying bias in detail for transparency and robustness. We extend the analysis to a total of 4 datasets and discuss the best practice in synthetic data curation for training indoor monocular depth. Further, dataset ablation is conducted to find out the key factor in generalization. This work marks the first in-depth investigation of performance variances across space types and, more importantly, releases useful tools, including datasets and codes, to closely examine your pretrained depth models. Data and code: https://depthcomputation.github.io/DepthPublic/

Figures (4)

• Figure 1: InSpaceType data samples. We color depth within 5 meters to show the near field. The captured data are dense, high-quality, and high-resolution for modern applications.
• Figure 2: Statistics of InSpaceType Evaluation Set. Note that the type percentage affects the overall results in Table \ref{['table:2']} but does not affect space type breakdowns in Table \ref{['table:25']} and \ref{['table:3']}, as each type performance is computed by averaging across all samples in a type and compared to other types.
• Figure 3: Zero-shot comparison between training strategies. The rooms are in oriental styles where the wall-hanging phone and air-conditioner are mostly exclusive to the cultural style. Compared with training on NYUv2 only, the multiple dataset pretraining shows higher robustness to unseen spaces, even though DPT is inferior to PixelFormer and Decompose on the standard NYUv2 dataset. Zero-shot evaluation on InSpaceType helps figure out advantages that cannot be shown if focusing on the single benchmark.
• Figure 4: Real kitchen scenes contain many small objects in large areas. However, the synthetic data cannot match the complexity hindering synthetic-to-real performance.