Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tiny-DroNeRF: Tiny Neural Radiance Fields aboard Federated Learning-enabled Nano-drones

Ilenia Carboni, Elia Cereda, Lorenzo Lamberti, Daniele Malpetti, Francesco Conti, Daniele Palossi

TL;DR

This work introduces Tiny-DroNeRF, a lightweight NeRF model, based on Instant-NGP, and optimized for running on a GAP9 ultra-low-power (ULP) MCU aboard nano-drones, and combines, for the first time, NeRF training on an ULP MCU with federated learning on nano-drones.

Abstract

Sub-30g nano-sized aerial robots can leverage their agility and form factor to autonomously explore cluttered and narrow environments, like in industrial inspection and search and rescue missions. However, the price for their tiny size is a strong limit in their resources, i.e., sub-100 mW microcontroller units (MCUs) delivering $\sim$100 GOps/s at best, and memory budgets well below 100 MB. Despite these strict constraints, we aim to enable complex vision-based tasks aboard nano-drones, such as dense 3D scene reconstruction: a key robotic task underlying fundamental capabilities like spatial awareness and motion planning. Top-performing 3D reconstruction methods leverage neural radiance fields (NeRF) models, which require GBs of memory and massive computation, usually delivered by high-end GPUs consuming 100s of Watts. Our work introduces Tiny-DroNeRF, a lightweight NeRF model, based on Instant-NGP, and optimized for running on a GAP9 ultra-low-power (ULP) MCU aboard our nano-drones. Then, we further empower our Tiny-DroNeRF by leveraging a collaborative federated learning scheme, which distributes the model training among multiple nano-drones. Our experimental results show a 96% reduction in Tiny-DroNeRF's memory footprint compared to Instant-NGP, with only a 5.7 dB drop in reconstruction accuracy. Finally, our federated learning scheme allows Tiny-DroNeRF to train with an amount of data otherwise impossible to keep in a single drone's memory, increasing the overall reconstruction accuracy. Ultimately, our work combines, for the first time, NeRF training on an ULP MCU with federated learning on nano-drones.

Tiny-DroNeRF: Tiny Neural Radiance Fields aboard Federated Learning-enabled Nano-drones

TL;DR

This work introduces Tiny-DroNeRF, a lightweight NeRF model, based on Instant-NGP, and optimized for running on a GAP9 ultra-low-power (ULP) MCU aboard nano-drones, and combines, for the first time, NeRF training on an ULP MCU with federated learning on nano-drones.

Abstract

Sub-30g nano-sized aerial robots can leverage their agility and form factor to autonomously explore cluttered and narrow environments, like in industrial inspection and search and rescue missions. However, the price for their tiny size is a strong limit in their resources, i.e., sub-100 mW microcontroller units (MCUs) delivering 100 GOps/s at best, and memory budgets well below 100 MB. Despite these strict constraints, we aim to enable complex vision-based tasks aboard nano-drones, such as dense 3D scene reconstruction: a key robotic task underlying fundamental capabilities like spatial awareness and motion planning. Top-performing 3D reconstruction methods leverage neural radiance fields (NeRF) models, which require GBs of memory and massive computation, usually delivered by high-end GPUs consuming 100s of Watts. Our work introduces Tiny-DroNeRF, a lightweight NeRF model, based on Instant-NGP, and optimized for running on a GAP9 ultra-low-power (ULP) MCU aboard our nano-drones. Then, we further empower our Tiny-DroNeRF by leveraging a collaborative federated learning scheme, which distributes the model training among multiple nano-drones. Our experimental results show a 96% reduction in Tiny-DroNeRF's memory footprint compared to Instant-NGP, with only a 5.7 dB drop in reconstruction accuracy. Finally, our federated learning scheme allows Tiny-DroNeRF to train with an amount of data otherwise impossible to keep in a single drone's memory, increasing the overall reconstruction accuracy. Ultimately, our work combines, for the first time, NeRF training on an ULP MCU with federated learning on nano-drones.
Paper Structure (18 sections, 3 equations, 6 figures, 2 tables)

This paper contains 18 sections, 3 equations, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Novel view synthesis methods
  4. Federated learning for neural radiance fields
  5. Background
  6. NeRF overview
  7. Robotic platform
  8. Methodology
  9. Hardware-aware memory-efficient NeRF
  10. NeRF deployment
  11. Federated implementation
  12. Experimental results
  13. Experimental setup
  14. Hardware-aware memory-efficient NeRF
  15. Deployment
  16. ...and 3 more sections

Figures (6)

  • Figure 1: A) Novel view synthesis with neural radiance fields and volumetric rendering mildenhall2020nerf. B) The InstantNGP algorithm.
  • Figure 2: Target robot platform. A) Bitcraze Crazyflie 2.1 Brushless with GAP9Shield expansion deck. B) GWT GAP9 System-on-Chip architecture.
  • Figure 3: Memory vs. reconstruction quality Pareto curve. Each point corresponds to a configuration of batch size $B$ (marker) and encoding size $\log_2(T)$ (number). Circled in red, the configuration selected for deployment.
  • Figure 4: Kernel latencies and peak on-chip memory usage per training step over a 1024-sample batch tile.
  • Figure 5: A) In-field data collection setup. B) Samples of ground-truth images and the respective reconstructions.
  • ...and 1 more figures