CPO: Change Robust Panorama to Point Cloud Localization

TL;DR

CPO addresses panorama-to-point-cloud localization under scene changes by combining fast color-histogram matching with robust 2D $M_{2D}$ and 3D $M_{3D}$ score maps. It avoids heavy per-view rendering through cached histograms and nearest-neighbor reuse, exploiting spherical projection to quickly generate millions of synthetic views. Candidate poses are efficiently initialized using histogram-based scoring and refined via a weighted loss $L_ ext{sampling}$ that downweights regions likely affected by changes, yielding stable convergence without pose priors. Experiments on OmniScenes, Structured3D, Stanford 2D-3D-S, and Data61/2D3D demonstrate strong change-robust localization, superior to baselines in both changed and unchanged scenes, with practical runtime on a single GPU. The approach offers a lightweight, scalable solution for AR/VR and autonomous navigation where scene changes are common.

Abstract

We present CPO, a fast and robust algorithm that localizes a 2D panorama with respect to a 3D point cloud of a scene possibly containing changes. To robustly handle scene changes, our approach deviates from conventional feature point matching, and focuses on the spatial context provided from panorama images. Specifically, we propose efficient color histogram generation and subsequent robust localization using score maps. By utilizing the unique equivariance of spherical projections, we propose very fast color histogram generation for a large number of camera poses without explicitly rendering images for all candidate poses. We accumulate the regional consistency of the panorama and point cloud as 2D/3D score maps, and use them to weigh the input color values to further increase robustness. The weighted color distribution quickly finds good initial poses and achieves stable convergence for gradient-based optimization. CPO is lightweight and achieves effective localization in all tested scenarios, showing stable performance despite scene changes, repetitive structures, or featureless regions, which are typical challenges for visual localization with perspective cameras. Code is available at \url{https://github.com/82magnolia/panoramic-localization/}.

Figures (10)

• Figure 1: Overview of our approach. CPO first creates 2D and 3D score maps that attenuate regions containing scene changes. The score maps are further used to guide candidate pose selection and pose refinement.
• Figure 2: Qualitative results of CPO. We show the query image (top), and the projected point cloud on the estimated pose (bottom). CPO can flexibly operate using raw color measurements or semantic labels.
• Figure 3: Illustration of fast histogram generation. For each image patch in the novel view $I_n$, we first project the patch centroid $c_i^n$ to the view of the original image $I_o$. The color histogram of the patch in the novel view is estimated as the histogram of image patch $c_*$ in the original view that is closest to the transformed centroid $p_i$.
• Figure 4: Illustration of 2D score map generation. The 2D score map for the $i$th patch $M_i$ is the maximum histogram intersection between the $i$th patch in query image $I_Q$ and the synthetic views $Y_n \in \mathcal{Y}$.
• Figure 5: Illustration of 3D score map generation. For each synthetic view $Y \in \mathcal{Y}$, the patch-wise color histogram is compared against the query image and the resulting intersection scores are back-projected onto 3D locations. The back-projected scores $B_Y$ are averaged for all synthetic views to form the 3D score map $M_\text{3D}$.