Adaptive Agent Selection and Interaction Network for Image-to-point cloud Registration

TL;DR

This work tackles image-to-point cloud registration by addressing cross-modal noise through an Adaptive Agent Selection and Interaction Network (A2SI). It introduces a Phase Map Extractor and Iterative Agents Selection (IAS) with a Tri-Stage optimization to identify reliable cross-modal agents, followed by Reliable Agents Interaction (RAI) that performs agent-guided cross-modal fusion, replacing standard dense transformer attention. The approach achieves state-of-the-art results on RGB-D Scenes v2 and 7-Scenes, with substantial gains in registration recall and robust performance in challenging scenes. The contributions offer improved accuracy and robustness for cross-modal registration and demonstrate effective reductions in attention noise and computation via agent-guided aggregation.

Abstract

Typical detection-free methods for image-to-point cloud registration leverage transformer-based architectures to aggregate cross-modal features and establish correspondences. However, they often struggle under challenging conditions, where noise disrupts similarity computation and leads to incorrect correspondences. Moreover, without dedicated designs, it remains difficult to effectively select informative and correlated representations across modalities, thereby limiting the robustness and accuracy of registration. To address these challenges, we propose a novel cross-modal registration framework composed of two key modules: the Iterative Agents Selection (IAS) module and the Reliable Agents Interaction (RAI) module. IAS enhances structural feature awareness with phase maps and employs reinforcement learning principles to efficiently select reliable agents. RAI then leverages these selected agents to guide cross-modal interactions, effectively reducing mismatches and improving overall robustness. Extensive experiments on the RGB-D Scenes v2 and 7-Scenes benchmarks demonstrate that our method consistently achieves state-of-the-art performance.

Figures (7)

• Figure 1: (a) Visual comparison of common and reliable agents correlations (b) Visualization of phase map enhancement. (c) Procedure and challenges of the reliable agents interaction method.
• Figure 2: Overall pipeline of A2SI. A phase map extractor first enhances image feature sensitivity to edge structures. The Tri-Stage Agents Optimization strategy then selects informative reliable agents from redundant learnable queries. These agents guide cross-modal aggregation in the Reliable Agents Interaction Module, progressively establishing dense correspondences in a coarse-to-fine manner. Finally, a rigid transformation is estimated via PnP with RANSAC.
• Figure 3: Visualization of difference between common transformer interaction and reliable agents interaction.
• Figure 4: Visualization of difference in (a) agents number (b) Convergence (c) t-sne.
• Figure 5: Visualization of matching accuracy: correct (green) if error $< 60$ px; otherwise red.