Object-level Geometric Structure Preserving for Natural Image Stitching

TL;DR

OBJ-GSP tackles natural image stitching by preserving object-level geometry while maintaining pixel alignment. It fuses semantic segmentation to extract object contours and triangle-mesh deformations with a four-term energy: $\psi_a(V)$, $\psi_l(V)$, $\psi_g(V)$, and $\psi_{obj}(V)$, optimized via linear least squares to balance alignment and distortion. The paper introduces StitchBench, a large diverse dataset for stitching, and demonstrates superior performance over grid-based and deep-learning baselines on distortion and naturalness metrics across multiple scenarios, including low-altitude aerial imagery. The findings suggest that object-level semantic information is essential for maintaining cohesive structures in challenging scenes, enabling broader practical applications.

Abstract

The topic of stitching images with globally natural structures holds paramount significance, with two main goals: pixel-level alignment and distortion prevention. The existing approaches exhibit the ability to align well, yet fall short in maintaining object structures. In this paper, we endeavour to safeguard the overall OBJect-level structures within images based on Global Similarity Prior (OBJ-GSP), on the basis of good alignment performance. Our approach leverages semantic segmentation models like the family of Segment Anything Model to extract the contours of any objects in a scene. Triangular meshes are employed in image transformation to protect the overall shapes of objects within images. The balance between alignment and distortion prevention is achieved by allowing the object meshes to strike a balance between similarity and projective transformation. We also demonstrate that object-level semantic information is necessary in low-altitude aerial image stitching. Additionally, we propose StitchBench, the largest image stitching benchmark with most diverse scenarios. Extensive experimental results demonstrate that OBJ-GSP outperforms existing methods in both pixel alignment and shape preservation. Code and dataset is publicly available at \url{https://github.com/RussRobin/OBJ-GSP}.

Figures (22)

• Figure 1: Red boxes indicate blurriness. (a) and (b) are not aligned well. (c) GSP gsp aligns well but distorts the building. Based on this, (d) GES-GSP gesgsp tries to prevent distortion but still fails in this case. (f) our method protects the structure of the building by sampling on object contours extracted by segmentation (e).
• Figure 2: Sampling points in our OBJ-GSP focus more on main structures so we can stitch precisely, as shown in (b)(d).
• Figure 3: (a) Our triangle mesh. $V_0$ is the center of object. (b) With every edge undergoing similarity and projection transformation, object in (a) is transformed into (b). (c) Triangle mesh with near-equilateral triangles of similar sizes across the region. (d) Triangle sampling strategy.
• Figure 4: APAP and SPHP see misalignment, and we delineated the indistinct portions using color-coded boxes. Autostitch, APAP, SPHP and GSP sees distortion. The convergence of the blue and red lines is essential, and we signify distortion by the intersection of these two lines. GES-GSP successfully prevents distortion, but it undergoes misalignment. Our method addresses misalignment and distortion well.
• Figure 5: We magnify the ground in the red box and the car in the purple box. OBJ-GSP aligns well but the remaining five methods shows misalignment. Distortion is not observed in this case.