Pose Optimization for Autonomous Driving Datasets using Neural Rendering Models

TL;DR

The paper tackles the problem of pose and calibration inaccuracies in public autonomous-driving datasets that can bias evaluation and downstream tasks. It introduces a unified NeRF-based optimization framework that jointly refines extrinsic calibration and vehicle trajectories, leveraging MOISST and SOAC alongside Nerfstudio baselines. Through comprehensive evaluation on KITTI-360, NuScenes, PandaSet, and Waymo Open, it demonstrates that optimized poses improve reprojection accuracy, novel view synthesis quality, and geometric consistency, with MOISST often outperforming alternatives. The work also provides a public release of optimized poses, underscoring its practical impact on improving dataset reliability and enabling more robust perception, mapping, and localization research.

Abstract

Autonomous driving systems rely on accurate perception and localization of the ego car to ensure safety and reliability in challenging real-world driving scenarios. Public datasets play a vital role in benchmarking and guiding advancement in research by providing standardized resources for model development and evaluation. However, potential inaccuracies in sensor calibration and vehicle poses within these datasets can lead to erroneous evaluations of downstream tasks, adversely impacting the reliability and performance of the autonomous systems. To address this challenge, we propose a robust optimization method based on Neural Radiance Fields (NeRF) to refine sensor poses and calibration parameters, enhancing the integrity of dataset benchmarks. To validate improvement in accuracy of our optimized poses without ground truth, we present a thorough evaluation process, relying on reprojection metrics, Novel View Synthesis rendering quality, and geometric alignment. We demonstrate that our method achieves significant improvements in sensor pose accuracy. By optimizing these critical parameters, our approach not only improves the utility of existing datasets but also paves the way for more reliable autonomous driving models. To foster continued progress in this field, we make the optimized sensor poses publicly available, providing a valuable resource for the research community.

Figures (14)

• Figure 1: Dataset pose improvement: Left, rendering comparison between original poses and optimized poses on Waymo open dataset sun2020scalability (from left to right: normal map, depth map and RGB rendering). Right, the changes in metrics between the original poses (in blue) and the poses optimized with MOISST (in red) for each dataset, grouped in 3 categories: Novel View Synthesis, Structure-from-Motion, Geometric.
• Figure 2: Pipeline: First the data is preprocessed to remove dynamic elements, and create subsequences fitting our NeRF models. Then, the extrinsic parameters are optimized, followed by the trajectories. Finally, the new poses are evaluated.
• Figure 3: NuScenes NeRF-LiDAR Novel View Synthesis
• Figure 4: NuScenes NeRF-LiDAR Novel View Synthesis normal maps
• Figure 5: PandaSet Nerfstudio Novel View Synthesis