FlyNeRF: NeRF-Based Aerial Mapping for High-Quality 3D Scene Reconstruction

TL;DR

FlyNeRF combines UAV-based data capture with a NeRF reconstruction framework and a CNN-driven render-quality assessor to iteratively improve 3D scene reconstructions in unknown environments. By using the image-evaluation module to identify low-quality renders and autonomously plan additional captures, the system enhances fidelity, achieving a 97% accuracy in quality prediction and a 2.5 dB PSNR improvement at the 10th percentile. Two-flight experiments demonstrate robust render-quality discrimination and significant gains in SSIM/PSNR, underscoring the method's potential for high-fidelity environmental mapping, monitoring, and digital twin applications. This approach enables more reliable 3D reconstructions for dynamic tasks such as navigation, surveillance, and infrastructure inspection, with practical benefits in autonomy and data efficiency.

Abstract

Current methods for 3D reconstruction and environmental mapping frequently face challenges in achieving high precision, highlighting the need for practical and effective solutions. In response to this issue, our study introduces FlyNeRF, a system integrating Neural Radiance Fields (NeRF) with drone-based data acquisition for high-quality 3D reconstruction. Utilizing unmanned aerial vehicle (UAV) for capturing images and corresponding spatial coordinates, the obtained data is subsequently used for the initial NeRF-based 3D reconstruction of the environment. Further evaluation of the reconstruction render quality is accomplished by the image evaluation neural network developed within the scope of our system. According to the results of the image evaluation module, an autonomous algorithm determines the position for additional image capture, thereby improving the reconstruction quality. The neural network introduced for render quality assessment demonstrates an accuracy of 97%. Furthermore, our adaptive methodology enhances the overall reconstruction quality, resulting in an average improvement of 2.5 dB in Peak Signal-to-Noise Ratio (PSNR) for the 10% quantile. The FlyNeRF demonstrates promising results, offering advancements in such fields as environmental monitoring, surveillance, and digital twins, where high-fidelity 3D reconstructions are crucial.

Figures (7)

• Figure 1: FlyNeRF system during the mission. The dashed red line with dots represents the trajectory executed by the drone and positions for additional image capture. The green area signifies the improvement in the quality of the reconstruction.
• Figure 2: FlyNeRF system architecture. The grey pointers represent the initial dataset capture during the first iteration of the mission for 3D reconstruction. The red pointer denotes the second iteration dedicated to capturing additional images for enhancing reconstruction quality.
• Figure 3: Image Evaluation Module architecture.
• Figure 4: Dataset collection pipeline for the training of CNN-based image evaluation.
• Figure 5: Render comparison after (a-c) the first and (d-f) the second iteration of the mission.