FaSS-MVS -- Fast Multi-View Stereo with Surface-Aware Semi-Global Matching from UAV-borne Monocular Imagery

TL;DR

In a thorough quantitative and ablative study, it is shown that the accuracy of the 3D information computed by FaSS-MVS is close to that of state-of-the-art offline multi-view stereo approaches, with the error not even an order of magnitude higher than that of COLMAP.

Abstract

With FaSS-MVS, we present an approach for fast multi-view stereo with surface-aware Semi-Global Matching that allows for rapid depth and normal map estimation from monocular aerial video data captured by UAVs. The data estimated by FaSS-MVS, in turn, facilitates online 3D mapping, meaning that a 3D map of the scene is immediately and incrementally generated while the image data is acquired or being received. FaSS-MVS is comprised of a hierarchical processing scheme in which depth and normal data, as well as corresponding confidence scores, are estimated in a coarse-to-fine manner, allowing to efficiently process large scene depths which are inherent to oblique imagery captured by low-flying UAVs. The actual depth estimation employs a plane-sweep algorithm for dense multi-image matching to produce depth hypotheses from which the actual depth map is extracted by means of a surface-aware semi-global optimization, reducing the fronto-parallel bias of SGM. Given the estimated depth map, the pixel-wise surface normal information is then computed by reprojecting the depth map into a point cloud and calculating the normal vectors within a confined local neighborhood. In a thorough quantitative and ablative study we show that the accuracies of the 3D information calculated by FaSS-MVS is close to that of state-of-the-art approaches for offline multi-view stereo, with the error not even being one magnitude higher than that of COLMAP. At the same time, however, the average run-time of FaSS-MVS to estimate a single depth and normal map is less than 14 % of that of COLMAP, allowing to perform an online and incremental processing of Full-HD imagery at 1-2 Hz.

Figures (12)

• Figure 1: Overview of the proposed approach for incremental MVS with plane-sweep multi-image matching and surface-aware SGM optimization. Given a bundle of images and corresponding camera poses $\left(\mathcal{I}, \mathbf{P}\right)_k$ of an input sequence, a hierarchical MVS estimation is performed to recover a depth, normal and confidence map $\left(\mathcal{D}, \mathcal{N}, \mathcal{C}\right)$.
• Figure 2: Overview of the algorithm for plane-sweep multi-image matching. A scene is sampled by a plane $\Pi = (\mathrm{n}, \delta)$, with $\mathrm{n}$ being the normal vector of the plane and $\delta$ being the orthogonal distance of the plane from $\mathrm{C}_{\mathrm{ref}}$, that is swept along its normal vector between two bounding planes $\Pi_{\mathrm{max}}$ and $\Pi_{\mathrm{min}}$ through space. For each distance $\delta$ of $\Pi$ the reference pixel $\mathrm{p}^{\mathrm{ref}}$ is projected by the plane induced homography $\mathbf{H}_{\mathrm{ref}\rightarrow k}$ into arbitrary number of viewpoints, where it is matched against the corresponding pixel in $\mathcal{I}_k$.
• Figure 3: Determination of the orthogonal distance parameter of the sampling planes of the plane-sweep multi-image matching by using the cross-ratio and epipolar geometry. Here, $\mathrm{C}_{\mathrm{ref}}$ and $\mathrm{C}_k$ represent the positions of the optical centers of the two cameras.
• Figure 4: Illustration of the three presented SGM$^{\mathrm{x}}$ path aggregations along one path direction $\mathrm{r}$. Column 1: Reference image and normal map of a building. Illustrated area marked with yellow line. Column 2: SGM$^{\Pi}$ path aggregation. The blue and pink lines represent the blue and pink surface orientations on the building facade. Aggregating the path costs for pixel $\mathrm{p}$ at plane $\Pi$, SGM$^{\Pi}$ will incorporate the previous costs at the same plane position (green) without additional penalty. The previous path costs at $\Pi\,\pm 1$ (yellow) will be penalized with $\varphi_1$. The previous path costs located at $\Pi\text{+}2$ (red), which is actually located on the corresponding surface, will be penalized with the highest penalty $\varphi_2$. Column 3: SGM$^{\Pi\text{-}\mathrm{sn}}$ uses the normal vector $\mathrm{n_p}$, encoding the surface orientation at pixel $\mathrm{p}$, and computes a discrete index jump $\Delta i_{\mathrm{sn}}$, which ideally adjusts the zero-cost transition, causing the previous path costs at $\Pi\text{+}2$ to not be penalized. Column 4: Similar to SGM$^{\Pi\text{-}\mathrm{sn}}$, SGM$^{\Pi\text{-}\mathrm{pg}}$ adjusts the zero-cost transition. However, the discrete index jump $\Delta i_{\mathrm{pg}}$ is derived from the running gradient $\nabla \mathrm{r}$ of the minimum cost path.
• Figure 5: Overview of the four datasets used for performance evaluation in the scope of this paper. Column 1: Two building models from the DTU Robot MVS dataset. Column 2: Example images in oblique and nadir view from the 3DOMcity Benchmark dataset. Column 3: Excerpt of the privately acquired TMB dataset. Column 4: Use-case-specific dataset acquired during an exercise of the local fire brigade.