Generative Point Cloud Registration

TL;DR

This work addresses the challenge of improving geometry-only point cloud registration by leveraging generative 2D models to synthesize cross-view RGB image pairs aligned with input clouds. It introduces Match-ControlNet, a depth-conditioned, matching-specific diffusion model that enforces 2D-3D geometric consistency and cross-view texture consistency, enabling geometric-color descriptor fusion. The approach supports zero-shot and few-shot settings, with two fusion schemes (zero-shot geometric-color fusion and XYZ-RGB fusion) that enhance robustness against calibration and lighting issues. Extensive experiments on ScanNet and 3DMatch demonstrate consistent improvements across multiple descriptors, validating the framework's plug-and-play applicability and effectiveness in challenging registration scenarios.

Abstract

In this paper, we propose a novel 3D registration paradigm, Generative Point Cloud Registration, which bridges advanced 2D generative models with 3D matching tasks to enhance registration performance. Our key idea is to generate cross-view consistent image pairs that are well-aligned with the source and target point clouds, enabling geometry-color feature fusion to facilitate robust matching. To ensure high-quality matching, the generated image pair should feature both 2D-3D geometric consistency and cross-view texture consistency. To achieve this, we introduce Match-ControlNet, a matching-specific, controllable 2D generative model. Specifically, it leverages the depth-conditioned generation capability of ControlNet to produce images that are geometrically aligned with depth maps derived from point clouds, ensuring 2D-3D geometric consistency. Additionally, by incorporating a coupled conditional denoising scheme and coupled prompt guidance, Match-ControlNet further promotes cross-view feature interaction, guiding texture consistency generation. Our generative 3D registration paradigm is general and could be seamlessly integrated into various registration methods to enhance their performance. Extensive experiments on 3DMatch and ScanNet datasets verify the effectiveness of our approach.

Figures (9)

• Figure 1: Paradigm comparison of our generative point cloud registration with conventional methods. Unlike geometry-only matching in previous methods, our approach introduces Match-ControlNet, a matching-specific 2D generative model that generates cross-view images pairs from point cloud data, providing rich color cues for enhanced geometric matching and pose estimation.
• Figure 2: Pipeline of Generative Point Cloud Registration. Given a source and a target point cloud, we first apply Match-ControlNet to generate their corresponding images. Next, we employ either zero-shot geometric-color feature fusion or XYZ-RGB fusion to create color-enhanced geometric descriptors, enabling high-quality correspondence estimation and robust pose estimation.
• Figure 3: Instead of independently performing ControlNet to generate source and target images, our Match-ControlNet integrates their denoising generation processes into a unified framework, facilitating feature interaction (i.e., mutual texture message passing) and enhancing their cross-view texture consistency.
• Figure 4: Compared to the zero-shot Match-ControlNet (top), the finetuned Match-ControlNet can tend to achieve higher 2D-3D geometric consistency and the cross-view texture consistency.
• Figure 5: Left: The visualization of the generated RGB image pairs and the formed color source and target point clouds; Right: In low-overlap cases, the original Predator struggles with registration. By contrast, the Generative Predator, enhanced with generated color information, successfully align them well.