Natural Image Stitching Using Depth Maps

TL;DR

A novel image stitching method using depth maps, which generates accurate alignment mosaics against parallax, and the depth maps of input images enable this method to provide much more accurate alignment in the overlapping region and view-consistent results in the non-overlapping region.

Abstract

Natural image stitching aims to create a single, natural-looking mosaic from overlapped images that capture the same 3D scene from different viewing positions. Challenges inevitably arise when the scene is non-planar and captured by handheld cameras since parallax is non-negligible in such cases. In this paper, we propose a novel image stitching method using depth maps, which generates accurate alignment mosaics against parallax. Firstly, we construct a robust fitting method to filter out the outliers in feature matches and estimate the epipolar geometry between input images. Then, we utilize epipolar geometry to establish pixel-to-pixel correspondences between the input images and render the warped images using the proposed optimal warping. In the rendering stage, we introduce several modules to solve the mapping artifacts in the warping results and generate the final mosaic. Experimental results on three challenging datasets demonstrate that the depth maps of input images enable our method to provide much more accurate alignment in the overlapping region and view-consistent results in the non-overlapping region. We believe our method will continue to work under the rapid progress of monocular depth estimation. The source code is available at https://github.com/tlliao/NIS_depths.

Figures (6)

• Figure 1: Stitching results of one test case in dataset zhang2014parallax via various methods. All results are generated via simple average blending, except that (l) is the warped target image via our method (best view in color and zoom in).
• Figure 2: Pipeline of the proposed image stitching method.
• Figure 3: Comparison of box plot distributions for different robust fitting methods tested on three datasets zhang2014parallaxlin2016seagullherrmann2018robust. From left to right: The number of feature matches, mapping error, and elapsed time. We test the three methods under different distance threshold settings in RANSAC and record the average values. All the mapping errors are calculated based on Eq. (\ref{['eq:maperror']}).
• Figure 4: Comparison of different image warping strategies on one test case. For clarity, we only draw 20$\times$30 mesh grids in (b) and (c).
• Figure 5: Comparison of the image stitching results obtained by our method with that of the four state-of-the-art existing methods: APAP zaragoza2014projective, GSP chen2016natural, REW li2018parallax, and MHW liao2025parallax (best view in color and zoom in).