MRASfM: Multi-Camera Reconstruction and Aggregation through Structure-from-Motion in Driving Scenes

TL;DR

MRASfM tackles the challenges of applying Structure-from-Motion to driving scenes captured by multi-camera rigs by introducing camera-set priors, semantic-aware road triangulation, and a transformation-based, two-tier bundle adjustment. The framework treats the multi-camera setup as a single rigid unit to improve pose reliability and computational efficiency, and uses plane fitting to suppress road-surface outliers. It further enables coarse-to-fine multi-scene aggregation to fuse fragmented drives into a coherent map. Real-world experiments and large-scale benchmarks (KITTI and nuScenes) demonstrate state-of-the-art pose accuracy and superior reconstruction efficiency, indicating strong potential for HD mapping and autonomous-vehicle perception tasks.

Abstract

Structure from Motion (SfM) estimates camera poses and reconstructs point clouds, forming a foundation for various tasks. However, applying SfM to driving scenes captured by multi-camera systems presents significant difficulties, including unreliable pose estimation, excessive outliers in road surface reconstruction, and low reconstruction efficiency. To address these limitations, we propose a Multi-camera Reconstruction and Aggregation Structure-from-Motion (MRASfM) framework specifically designed for driving scenes. MRASfM enhances the reliability of camera pose estimation by leveraging the fixed spatial relationships within the multi-camera system during the registration process. To improve the quality of road surface reconstruction, our framework employs a plane model to effectively remove erroneous points from the triangulated road surface. Moreover, treating the multi-camera set as a single unit in Bundle Adjustment (BA) helps reduce optimization variables to boost efficiency. In addition, MRASfM achieves multi-scene aggregation through scene association and assembly modules in a coarse-to-fine fashion. We deployed multi-camera systems on actual vehicles to validate the generalizability of MRASfM across various scenes and its robustness in challenging conditions through real-world applications. Furthermore, large-scale validation results on public datasets show the state-of-the-art performance of MRASfM, achieving 0.124 absolute pose error on the nuScenes dataset.

Figures (11)

• Figure 1: The idea of MRASfM. To address the challenges of SfM in driving scenes, MRASfM incorporates camera set priors and semantic information into the reconstruction framework, achieving efficient and robust scene reconstruction. Through multi-scene aggregation, fragmented scenes are integrated into a complete and consistent map.
• Figure 2: The pipeline of MRASfM. MRASfM takes multi-camera images, semantic information, rough trajectories, and calibrations as input. When reconstructing a scene, MRASfM first utilizes prior information to perform correspondence search and initialization (see \ref{['subsec:Correspondence Search and Initialization']}). In camera set registration, MRASfM robustly registers new images with rigid unit constraints (see \ref{['subsec:RIG_Registratio']}). During triangulation, MRASfM improves road surface reconstruction quality using semantic information for fewer outliers (see \ref{['subsec:Semantic-assisted Triangulation']}). In camera set BA, MRASfM refines the reconstruction using rigid units as optimization variables, enhancing efficiency and robustness (see \ref{['subsec:RIG BA']}). When aggregating multiple scenes, MRASfM assembles nearby scenes with SfM in a coarse-to-fine fashion (see \ref{['subsec:Multi-scene Aggregation']}).
• Figure 3: The pipeline of Camera Set Registration: camera pose estimation via PnP solving, multi-view fused vehicle pose derivation, and camera pose updating using intra-unit constraints and vehicle poses.
• Figure 4: Comparison between traditional BA and Camera Set BA. Traditional BA optimizes the absolute camera pose of each image individually. Camera Set BA, by contrast, optimizes vehicle poses and internal relative poses, significantly reducing optimization variables and enhancing robustness.
• Figure 5: Pipeline of Scene Assembly. MRASfM begins by selecting images with rich correspondences for relocalization. By comparing the original and updated camera poses, the transformation matrix is initialized, which is then optimized by BA. The refined matrix is used to register remaining rigid units into reconstruction. Registration and refinement alternate until all rigid units are successfully integrated.