Learning a Depth Covariance Function

TL;DR

This work proposes learning a depth covariance function with applications to geometric vision tasks, which can be flexibly used to define priors over depth functions, predictive distributions given observations, and methods for active point selection.

Abstract

We propose learning a depth covariance function with applications to geometric vision tasks. Given RGB images as input, the covariance function can be flexibly used to define priors over depth functions, predictive distributions given observations, and methods for active point selection. We leverage these techniques for a selection of downstream tasks: depth completion, bundle adjustment, and monocular dense visual odometry.

Figures (10)

• Figure 1: Example monocular reconstruction using the depth covariance for bundle adjustment and dense depth prediction from three seconds (100 frames) of TUM data. Three representative images and the mesh from TSDF fusion of the depth predictions are shown. Each frame leverages the learned covariance function to model geometric correlation between pairs of scene points.
• Figure 2: Visualizing our depth covariance function: for every pixel of an input image, the trained network predicts a 2D kernel matrix. Here we show the covariance function between pairs of pixels in both matrix form and as edges in a graph, with the line thickness representing the magnitude of covariance.
• Figure 3: Conditioning example for 128 samples. The posterior variance is high around edges and in areas lacking samples. The columns of $K_{\text{fn}}$, or correlation maps, are shown for select points.
• Figure 4: Qualitative comparison of random and active sampling given 32 sample selections. Random sampling misrepresents depth at the top of the image, while active sampling focuses on the chair geometry and avoids redundant samples on the floor.
• Figure 5: Calibration plots of varying posterior marginal covariance block dimensions $D$ on NYUv2 depth completion. The ideal calibration is $y=x$, where the observed confidence matches expected confidence. The region above the line indicates model under-confidence, while the area below signals over-confidence.