Pano360: Perspective to Panoramic Vision with Geometric Consistency

Abstract

Prior panorama stitching approaches heavily rely on pairwise feature correspondences and are unable to leverage geometric consistency across multiple views. This leads to severe distortion and misalignment, especially in challenging scenes with weak textures, large parallax, and repetitive patterns. Given that multi-view geometric correspondences can be directly constructed in 3D space, making them more accurate and globally consistent, we extend the 2D alignment task to the 3D photogrammetric space. We adopt a novel transformer-based architecture to achieve 3D awareness and aggregate global information across all views. It directly utilizes camera poses to guide image warping for global alignment in 3D space and employs a multi-feature joint optimization strategy to compute the seams. Additionally, to establish an evaluation benchmark and train our network, we constructed a large-scale dataset of real-world scenes. Extensive experiments show that our method significantly outperforms existing alternatives in alignment accuracy and perceptual quality.

Figures (6)

• Figure 1: Comparison in repetitive patterns. GES-GSP du2022geometric suffers from inaccurate pairwise feature correspondences, causing severe artifacts. Leveraging multi-view geometric consistency in 3D space effectively filters unreliable matches, enabling robust alignment. Our method achieves a 97.8% success rate in challenging scenes with weak textures, large parallax, and repetitive patterns.
• Figure 2: Architecture Overview. (a) Perspective images are projected onto a common panoramic coordinate system using camera parameters. (b) Overlapping regions are extracted from the globally aligned images (a single image may overlap multiple neighbors). (c) The seam decoder is supervised with three weight maps and generates the seam mask for each image. (d) The final panorama is blended using the seam mask and the globally aligned images, supporting various projection formats.
• Figure 3: Qualitative comparison of our method Pano360 to UDIS2 on challenging real-world scenarios. As shown in the first column, our method successfully handles the large parallax scene, while UDIS2 suffers from ghosting and misalignment. In the second column, our method stitches the images in the weak-textured fog scene, while UDIS2 fails. The third column presents a challenging case with repetitive patterns, unusual lighting, and a large FoV. Our method successfully recovers a panorama from 24 frames, whereas UDIS2 is limited to pairwise processing and suffers from accumulated errors, leading to severe geometric distortion and failure.
• Figure 4: Visualization of the proposed Pano360 dataset. It showcases distinct and challenging scenes. Each case presents several source images, their seam masks, and the final panorama we generated.
• Figure 5: Qualitative comparison of panorama stitching on challenging in-the-wild images. Existing methods suffer from noticeable artifacts and distortions on buildings with repetitive patterns. However, our method achieves globally consistent alignment and remains free of these artifacts.