Bridge 2D-3D: Uncertainty-aware Hierarchical Registration Network with Domain Alignment

TL;DR

Bridge 2D-3D proposes B2-3Dnet, an uncertainty-aware hierarchical registration network with domain alignment to tackle cross-modal 2D–3D patch matching. It introduces UHMM to weigh image patches by uncertainty across multiple scales and AMAM to reduce modality gaps via a gradient reversal-based adversarial objective. The method achieves state-of-the-art registration metrics on RGB-D Scene V2 and 7-Scenes, demonstrating improved inlier ratios, feature matching recall, and robust generalization to scale variations. By combining a hierarchical patch-level matching strategy with adversarial domain alignment and a PnP-RANSAC pose estimator, the approach yields accurate, robust image-to-point cloud registration suitable for 3D reconstruction and localization tasks.

Abstract

The method for image-to-point cloud registration typically determines the rigid transformation using a coarse-to-fine pipeline. However, directly and uniformly matching image patches with point cloud patches may lead to focusing on incorrect noise patches during matching while ignoring key ones. Moreover, due to the significant differences between image and point cloud modalities, it may be challenging to bridge the domain gap without specific improvements in design. To address the above issues, we innovatively propose the Uncertainty-aware Hierarchical Matching Module (UHMM) and the Adversarial Modal Alignment Module (AMAM). Within the UHMM, we model the uncertainty of critical information in image patches and facilitate multi-level fusion interactions between image and point cloud features. In the AMAM, we design an adversarial approach to reduce the domain gap between image and point cloud. Extensive experiments and ablation studies on RGB-D Scene V2 and 7-Scenes benchmarks demonstrate the superiority of our method, making it a state-of-the-art approach for image-to-point cloud registration tasks.

Figures (4)

• Figure 1: (a) Visualization of uncertainty modeling, I and P form a pair. Variance indicates uncertainty: better matches have lower uncertainty, while poor matches have higher uncertainty. (b) Visualization of modality differences. With training, the modalities and distributions of the point cloud and image become aligned.
• Figure 2: Overall pipeline of B2-3Dnet. We use a feature extraction backbone to obtain features from images and point clouds, which are aligned using the adversarial modal alignment module to reduce domain differences. The image features are processed through hierarchical layers and uncertainty estimation layers to create informative image patches. During the interaction stages, updated point cloud patches and image features generate a score map via cosine similarity and maximum, achieving coarse-level matching and refining fine-level matches. Finally, PnP+RANSAC is used to regress the rigid transformation.
• Figure 3: Comparison of the effects with or without an uncertainty estimation layer on correctly or incorrectly matched patches.
• Figure 4: (a) Ablation studies of our model. (b) Ablation studies on $\gamma$. (c) Ablation studies on $\lambda$.