Redefining Recon: Bridging Gaps with UAVs, 360 degree Cameras, and Neural Radiance Fields

TL;DR

The paper investigates rapid 3D reconstruction for disaster response by deploying ultra-small FPV UAVs with 360° cameras and training Neural Radiance Fields on the collected imagery, enabling free-view rendering of post-disaster scenes. NeRFs operate on a $5$-D input vector $(x,y,z, heta,phi)$ to encode geometry and viewing direction, providing high-fidelity representations that outperform traditional multi-view stereo under challenging lighting and reflective conditions. The study compares three pipelines—OpenDroneMap/WebODM, PatchMatch-Stereo-Panorama, and Nerfstudio/Nerfacto—showing NeRF-based reconstructions yield the most accurate novel views but require substantial compute resources. The end-to-end system demonstrates fast field applicability, surveying a $20$ m by $70$ m space in under 4 minutes and delivering a usable 3D model within about 15 minutes, offering tangible benefits for indoor GPS-denied and post-disaster rescue operations.

Abstract

In the realm of digital situational awareness during disaster situations, accurate digital representations, like 3D models, play an indispensable role. To ensure the safety of rescue teams, robotic platforms are often deployed to generate these models. In this paper, we introduce an innovative approach that synergizes the capabilities of compact Unmaned Arial Vehicles (UAVs), smaller than 30 cm, equipped with 360 degree cameras and the advances of Neural Radiance Fields (NeRFs). A NeRF, a specialized neural network, can deduce a 3D representation of any scene using 2D images and then synthesize it from various angles upon request. This method is especially tailored for urban environments which have experienced significant destruction, where the structural integrity of buildings is compromised to the point of barring entry-commonly observed post-earthquakes and after severe fires. We have tested our approach through recent post-fire scenario, underlining the efficacy of NeRFs even in challenging outdoor environments characterized by water, snow, varying light conditions, and reflective surfaces.

Figures (6)

• Figure 1: Examples of small UAVs. Top: DJI FPV adapted with one and two 360° cameras. Bottom: DJI Avata and a self-constructed invisible UAV, both with 360° cameras.
• Figure 2: Schematic drawing illustrating the functioning of the self-constructed invisible uav.
• Figure 3: Panoramas of two UAVs with 360° cameras.
• Figure 4: Diagram depicting data processing using different methods after the flight mission. Preprocessing steps are highlighted in yellow, main processing in blue, and the flight mission and results are denoted in red and green, respectively.
• Figure 5: Assessment of the self-constructed invisible UAV using different methods at the Westphalian University of Applied Sciences.