Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields

TL;DR

Depth estimation from a single RGB image is highly ill-posed. The authors propose a deep convolutional neural field (DCNF) that jointly learns unary potentials from CNNs and pairwise potentials over superpixels within a continuous CRF, enabling analytic likelihood optimization and closed-form MAP inference. They further develop DCNF-FCSP, a faster variant using fully convolutional networks and a novel superpixel pooling method, which enables deeper architectures without sacrificing accuracy. Empirical results on NYU v2, Make3D, and KITTI show state-of-the-art performance and substantial speedups, demonstrating the practical impact of combining deep learning with continuous CRFs for dense monocular depth estimation.

Abstract

In this article, we tackle the problem of depth estimation from single monocular images. Compared with depth estimation using multiple images such as stereo depth perception, depth from monocular images is much more challenging. Prior work typically focuses on exploiting geometric priors or additional sources of information, most using hand-crafted features. Recently, there is mounting evidence that features from deep convolutional neural networks (CNN) set new records for various vision applications. On the other hand, considering the continuous characteristic of the depth values, depth estimations can be naturally formulated as a continuous conditional random field (CRF) learning problem. Therefore, here we present a deep convolutional neural field model for estimating depths from single monocular images, aiming to jointly explore the capacity of deep CNN and continuous CRF. In particular, we propose a deep structured learning scheme which learns the unary and pairwise potentials of continuous CRF in a unified deep CNN framework. We then further propose an equally effective model based on fully convolutional networks and a novel superpixel pooling method, which is $\sim 10$ times faster, to speedup the patch-wise convolutions in the deep model. With this more efficient model, we are able to design deeper networks to pursue better performance. Experiments on both indoor and outdoor scene datasets demonstrate that the proposed method outperforms state-of-the-art depth estimation approaches.

Figures (12)

• Figure 1: Examples of depth estimation results using the proposed deep convolutional neural fields model. First row: NYU v2 dataset; second row: Make3D dataset. From left to right: input image, ground-truth, our prediction.
• Figure 2: An illustration of our DCNF model for depth estimation. The input image is first over-segmented into superpixels. In the unary part, for a superpixel $p$, we crop the image patch centred around its centroid, then resize and feed it to a CNN which is composed of 5 convolutional and 4 fully-connected layers (details refer to Fig. \ref{['fig:dcnf_unary']}). In the pairwise part, for a pair of neighboring superpixels $(p, q)$, we consider $K$ types of similarities, and feed them into a fully-connected layer. The outputs of unary part and the pairwise part are then fed to the CRF structured loss layer, which minimizes the negative log-likelihood. Predicting the depths of a new image ${\bf x}$ is to maximize the conditional probability ${\mathrm{Pr}}({\bf y}|{\bf x})$, which has closed-form solutions (see Sec. \ref{['sec:learning']} for details).
• Figure 3: Detailed network architecture of the unary part in Fig. \ref{['fig:dcnf']}.
• Figure 4: An overview of the unary part of the DCNF-FCSP model. For the unary part, the input image is fed into a fully-convolutional network to produce convolution maps ($d$ is the number of filters of the last fully-convolutional layer). The obtained convolution maps, together with the superpixel segmentation over the original input image, are fed to a superpixel pooling layer. The outputs are $n \times 1$$d$ dimensional feature vectors for each of the $n$ superpixels, which are then followed by $3$ fully-connected layers to produce the unary output ${\bf z}$. The pairwise part are omitted here since we use the same network architecture as in the DCNF model (Fig. \ref{['fig:dcnf']}). The unary output ${\bf z}$ and the pairwise output ${\bf R}$ are used as input to the CRF loss layer, which minimizes the negative log-likelihood (See Sec. \ref{['sec:dcnfsp']} for details) .
• Figure 5: The fully convolutional network architecture used in Fig. \ref{['fig:dcnfsp']}. The network takes input images of arbitrary size and output convolution maps.