Rendering-Enhanced Automatic Image-to-Point Cloud Registration for Roadside Scenes

TL;DR

This work tackles automatic registration between roadside camera images and prior point clouds by introducing neighbor rendering to generate view-aligned, correspondence-preserving images from sparse point clouds. A two-stage pipeline first uses rendered views and SuperGlue to obtain initial 2D-3D correspondences, then refines the extrinsics by minimizing line-based reprojection error with line features extracted by SAM, formulated in $SE(3)$ and solved via Ceres. Evaluations on a self-collected eight-site dataset show translation and rotation errors of approximately $0.079$ m and $0.202^\circ$, respectively, with substantial improvements in ground-distance accuracy and downstream monocular 3D object detection. The approach demonstrates robust, automated roadside registration that enables accurate fusion of 3D priors with monocular vision, offering practical benefits for traffic surveillance and autonomous roadside perception.

Abstract

Prior point cloud provides 3D environmental context, which enhances the capabilities of monocular camera in downstream vision tasks, such as 3D object detection, via data fusion. However, the absence of accurate and automated registration methods for estimating camera extrinsic parameters in roadside scene point clouds notably constrains the potential applications of roadside cameras. This paper proposes a novel approach for the automatic registration between prior point clouds and images from roadside scenes. The main idea involves rendering photorealistic grayscale views taken at specific perspectives from the prior point cloud with the help of their features like RGB or intensity values. These generated views can reduce the modality differences between images and prior point clouds, thereby improve the robustness and accuracy of the registration results. Particularly, we specify an efficient algorithm, named neighbor rendering, for the rendering process. Then we introduce a method for automatically estimating the initial guess using only rough guesses of camera's position. At last, we propose a procedure for iteratively refining the extrinsic parameters by minimizing the reprojection error for line features extracted from both generated and camera images using Segment Anything Model (SAM). We assess our method using a self-collected dataset, comprising eight cameras strategically positioned throughout the university campus. Experiments demonstrate our method's capability to automatically align prior point cloud with roadside camera image, achieving a rotation accuracy of 0.202 degrees and a translation precision of 0.079m. Furthermore, we validate our approach's effectiveness in visual applications by substantially improving monocular 3D object detection performance.

Figures (11)

• Figure 1: Our method registers drone-collected point clouds with roadside camera images. This registration enhances monocular algorithm performance for downstream roadside tasks.
• Figure 2: Our automatic registration method follows a two-step framework. (1) In initial guess estimation pipeline, we first sample camera poses around the rough location and synthesize views, then match this views with camera image using SuperGlue and estimate the initial guess by 2D-3D correspondence. (2) In optimization process, we first extract lines in point cloud and image, then we match these lines and optimize the extrinsics using reprojection errors.
• Figure 3: The purple points represent foreground points, while the green points represent background points. Neighbor rendering filters out the background points and estimate the 3D point $P_i$ corresponding to pixel $p_i$ based on the foreground points.
• Figure 4: The typical installation for roadside cameras. We assume that the camera has a small pitch angle and place pseudo-cameras at different yaw angles to generate views.
• Figure 5: Left is cones in camera images, right is the corresponding point clouds, the points size has been enlarged for better visualization.