LiftProj: Space Lifting and Projection-Based Panorama Stitching

TL;DR

This work addresses distortions and ghosting in panorama stitching caused by parallax and occlusions under traditional 2D warping. It proposes a three-stage pipeline that lifts input images to dense 3D point clouds, fuses them in a common 3D frame with confidence-guided weighting, and reprojects to a 360° panorama using a unified projection center with equidistant cylindrical projection, followed by canvas-domain hole completion. Key contributions include (1) shifting stitching from 2D warping to 3D geometric consistency, (2) introducing a unified projection center with cylindrical projection to reduce multi-view distortion, and (3) integrating a self-supervised MAE-based hole-filling module to restore unobserved regions. The approach demonstrates substantial reductions in distortion and ghosting in challenging parallax scenarios and delivers more natural, continuous panoramas, supported by experiments on the MVIS dataset and comparisons to state-of-the-art methods. This framework enables flexible incorporation of various 3D lifting and completion modules and has practical implications for robust 360° immersive imaging under complex scene geometry.

Abstract

Traditional image stitching techniques have predominantly utilized two-dimensional homography transformations and mesh warping to achieve alignment on a planar surface. While effective for scenes that are approximately coplanar or exhibit minimal parallax, these approaches often result in ghosting, structural bending, and stretching distortions in non-overlapping regions when applied to real three-dimensional scenes characterized by multiple depth layers and occlusions. Such challenges are exacerbated in multi-view accumulations and 360° closed-loop stitching scenarios. In response, this study introduces a spatially lifted panoramic stitching framework that initially elevates each input image into a dense three-dimensional point representation within a unified coordinate system, facilitating global cross-view fusion augmented by confidence metrics. Subsequently, a unified projection center is established in three-dimensional space, and an equidistant cylindrical projection is employed to map the fused data onto a single panoramic manifold, thereby producing a geometrically consistent 360° panoramic layout. Finally, hole filling is conducted within the canvas domain to address unknown regions revealed by viewpoint transitions, restoring continuous texture and semantic coherence. This framework reconceptualizes stitching from a two-dimensional warping paradigm to a three-dimensional consistency paradigm and is designed to flexibly incorporate various three-dimensional lifting and completion modules. Experimental evaluations demonstrate that the proposed method substantially mitigates geometric distortions and ghosting artifacts in scenarios involving significant parallax and complex occlusions, yielding panoramic results that are more natural and consistent.

Figures (10)

• Figure 1: Schematic diagram of the algorithm workflow presented in this paper. The 2D input images are first processed by a reconstruction network to generate corresponding 3D point cloud scenes. For multiple input images, the generated scenes are globally aligned. Then, an equidistant cylindrical projection is applied to reduce the dimensionality of the 3D point clouds onto a 2D manifold. Finally, hole filling is performed on the resulting images. In this paper, based on an improved MAE algorithm, the output of MAE is used as a prior input to the completion network, enabling large-scale semantic completion.
• Figure 2: Illustration of projection distortion. (a) Stitching result using Autostitch: the overlapping region exhibits minimal ghosting artifacts, whereas the non-overlapping region suffers from pronounced stretching distortion, as indicated by the red arrows. (b) Stitching result using UDIS++ 2023udis++: distortion in the non-overlapping region is reduced, albeit at the expense of conspicuous ghosting artifacts within the overlapping region.
• Figure 3: Visualization of projection-induced holes. The green box highlights regular holes attributable to discrete sampling, whereas the red box emphasizes larger holes resulting from changes in viewpoint and occlusion.
• Figure 4: Representative samples from the MVIS dataset. Panels (a)–(d) depict challenging scenes, whereas panels (e)–(f) illustrate simpler scenes.
• Figure 5: Comparison of stitching results in two-image scenarios on the MVIS dataset. Representative overlapping regions were cropped and magnified to illustrate alignment quality. Achieving simultaneous alignment of both near and distant objects in scenes with rich foreground and background content presents significant challenges. Autostitch 2007Automatic relies exclusively on a single homography matrix for global alignment, lacking local alignment capabilities. AANAP 2015Aanap employs local grid alignment, which improves local alignment relative to Autostitch but remains inadequate for aligning near and far objects in high-parallax scenes. GSP 2016gsp further refines local alignment but exhibits instability under large parallax conditions. Deep learning-based methods UDIS++ 2023udis++ and MHW 2024Parallax inherit the traditional paradigm of combining global and local alignment, yet still struggle to simultaneously align objects at varying depths in scenes with substantial parallax. Our approach reconstructs scene objects via dimensionality lifting and performs separate alignment of objects at different depths within 3D space, thereby effectively addressing the alignment of near and far objects in large parallax scenarios.