Blending Distributed NeRFs with Tri-stage Robust Pose Optimization

TL;DR

This work tackles large-scale urban NeRF reconstruction by addressing cross-NeRF registration and blending in distributed setups. It introduces a tri-stage pose optimization built on Mip-NeRF 360: (1) bundle-adjusting NeRF for joint scene- and pose-refinement, (2) Frame2Model optimization with iMNeRF and a truncated dynamic low-pass filter to robustly align NeRFs to a frame, and (3) Model2Model refinement to align multiple NeRFs in a common coordinate system before blending. The approach yields improved registration accuracy and blending quality, demonstrated on real and simulated datasets with favorable PSNR, SSIM, and LPIPS, and achieves compact model sizes. The public release of DUMAD provides resources to evaluate distributed NeRFs in city-scale scenes, highlighting the method's practical impact for scalable, photorealistic urban reconstruction and navigation pipelines.

Abstract

Due to the limited model capacity, leveraging distributed Neural Radiance Fields (NeRFs) for modeling extensive urban environments has become a necessity. However, current distributed NeRF registration approaches encounter aliasing artifacts, arising from discrepancies in rendering resolutions and suboptimal pose precision. These factors collectively deteriorate the fidelity of pose estimation within NeRF frameworks, resulting in occlusion artifacts during the NeRF blending stage. In this paper, we present a distributed NeRF system with tri-stage pose optimization. In the first stage, precise poses of images are achieved by bundle adjusting Mip-NeRF 360 with a coarse-to-fine strategy. In the second stage, we incorporate the inverting Mip-NeRF 360, coupled with the truncated dynamic low-pass filter, to enable the achievement of robust and precise poses, termed Frame2Model optimization. On top of this, we obtain a coarse transformation between NeRFs in different coordinate systems. In the third stage, we fine-tune the transformation between NeRFs by Model2Model pose optimization. After obtaining precise transformation parameters, we proceed to implement NeRF blending, showcasing superior performance metrics in both real-world and simulation scenarios. Codes and data will be publicly available at https://github.com/boilcy/Distributed-NeRF.

Figures (4)

• Figure 1: System Overview. Each agent trains a bundle-adjusting Mip-NeRF 360. Following this, co-view region retrieval provides an initial value for Frame2Model pose optimization. A final Model2Model pose optimization is performed before blending to achieve a seamless fusion of NeRFs.
• Figure 2: Different Dynamic Low-pass Filter.
• Figure 3: Frame2Model registration results: the LATITUDElatitude experiences registration failure due to aliasing, resulting in confusion in the merged observation and rendering images. While the iNGPliu2023baa shows notable accuracy improvements, noticeable floating artifacts in the highlighted area are observed, potentially inducing registration errors. Ours effectively avoids aliasing, ensuring superior rendering quality and registration accuracy.
• Figure 4: Blending Results in Real World. The first two columns represent the rendering results of distributed NeRFs trained by different methods. Due to data loss at the edges of the UAV trajectory, aliasing occurs in the rendering as indicated. When comparing renderings from the two blended NeRFs, our method significantly enhances areas where individual NeRF renderings perform poorly.