Parallax-tolerant Image Stitching via Segmentation-guided Multi-homography Warping

TL;DR

This work addresses large parallax in image stitching by introducing a segmentation-guided multi-homography warping framework. It leverages the Segment Anything Model (SAM) to partition the target image into semantic contents, then partitions feature matches into multiple homography models via an energy-based fitting process; overlapping contents are labeled with the best-fitting homography, while non-overlapping regions are extrapolated with linearly blended, linearized warps. Warping is performed in a forward-backward manner with an error-buffer to handle occlusions, followed by linear blending to produce the final panorama. Quantitative and qualitative results on public parallax datasets show substantial improvements in alignment quality over state-of-the-art methods. Overall, the approach advances parallax-robust stitching by combining semantic segmentation with content-aware, multi-homography modeling and occlusion-aware rendering.

Abstract

Large parallax between images is an intractable issue in image stitching. Various warping-based methods are proposed to address it, yet the results are unsatisfactory. In this paper, we propose a novel image stitching method using multi-homography warping guided by image segmentation. Specifically, we leverage the Segment Anything Model to segment the target image into numerous contents and partition the feature points into multiple subsets via the energy-based multi-homography fitting algorithm. The multiple subsets of feature points are used to calculate the corresponding multiple homographies. For each segmented content in the overlapping region, we select its best-fitting homography with the lowest photometric error. For each segmented content in the non-overlapping region, we calculate a weighted combination of the linearized homographies. Finally, the target image is warped via the best-fitting homographies to align with the reference image, and the final panorama is generated via linear blending. Comprehensive experimental results on the public datasets demonstrate that our method provides the best alignment accuracy by a large margin, compared with the state-of-the-art methods. The source code is available at https://github.com/tlliao/multi-homo-warp.

Figures (9)

• Figure 1: Stitching results comparison with various warping methods. All results are generated via simple linear blending. Obvious parallax can be seen from the tower and the red statue. (e)-(g) correspond to spatially-varying warps, (h)-(j) correspond to mesh-based warps, and (k) is the learning-based warp. Compared with other warps, our method achieves the best alignment quality.
• Figure 2: Pipeline of our multi-homography warping method.
• Figure 3: Partition results of the feature points by our multi-homography fitting algorithm w/o and w/ SAM. (a) Neighborhood system defined only by Delaunay triangulation and the partition result. (b) Neighborhood system defined by Delaunay triangulation & SAM, and the partition result. Introducing SAM to define the neighborhood system boosts the model performance.
• Figure 4: Labeling results on different image pairs. Top: SAM results. Middle: labeling results for overlapping regions. Each color corresponds to a label and fits a homography. Bottom: average photometric errors for each labeling content, where errors are shown as a hot map.
• Figure 5: Different Labeling strategies for the non-overlapping region. (a) Labeling strategy which replaces $\mathbf{S}$ with global homography $H_g$. (b) Labeling strategy only using the first term of Eq. \ref{['eq:nonoverlapping']}. (c) Labeling strategy which replaces dense cells with segmented contents by SAM. (d) Our final labeling strategy.