Fully Onboard Low-Power Localization with Semantic Sensor Fusion on a Nano-UAV using Floor Plans

TL;DR

This work tackles indoor localization for GPS-denied nano-UAVs under severe power and compute constraints by fusing geometric information from miniaturized ToF sensors with semantic cues obtained from an onboard object detector. The authors deploy a fully onboard, memory-efficient semantic map and Monte Carlo Localization on a GAP9 multi-core RISC-V processor, achieving real-time performance without external infrastructure. Key contributions include a quantized YOLOv5p object detector running at 20 fps with minimal memory and energy overhead, a compact 16-bit semantic map format, and a novel fusion model that improves global localization in floor-plan environments. The method demonstrates 90% localization success with an average error of 0.32 m in real-world office settings and is released as open source, enabling broader adoption for autonomous nano-UAVs.

Abstract

Nano-sized unmanned aerial vehicles (UAVs) are well-fit for indoor applications and for close proximity to humans. To enable autonomy, the nano-UAV must be able to self-localize in its operating environment. This is a particularly-challenging task due to the limited sensing and compute resources on board. This work presents an online and onboard approach for localization in floor plans annotated with semantic information. Unlike sensor-based maps, floor plans are readily-available, and do not increase the cost and time of deployment. To overcome the difficulty of localizing in sparse maps, the proposed approach fuses geometric information from miniaturized time-of-flight sensors and semantic cues. The semantic information is extracted from images by deploying a state-of-the-art object detection model on a high-performance multi-core microcontroller onboard the drone, consuming only 2.5mJ per frame and executing in 38ms. In our evaluation, we globally localize in a real-world office environment, achieving 90% success rate. We also release an open-source implementation of our work.

Figures (6)

• Figure 1: Top: A nano-UAV while flying and globally localizing in an office environment using our novel sensor fusion approach. Bottom: A qualitative evaluation of the localization results on recorded sequences. Ground truth pose is marked by black stars. The rainbow colors encode the time of prediction, with purple marking the beginning of the sequence and red its end.
• Figure 2: System overview. Top: All stacked components on the nano-UAV, ordered from the top (left) to the bottom (right). Bottom: A visualization of the communication paths and task distribution between all employed processors and sensors.
• Figure 3: Left: A top view of the dense pointcloud captured with the Z+F Imager 5016 terrestrial laser scanner, which was used solely for GT extraction. The full pointcloud has 200 million points. Right: The semantically-enriched floor plan of the lab. Semantic objects of interest are represented using their bounding box and class ID. Different colors represent different object classes. The semantic information was added manually, without a complex measuring or mapping procedure.
• Figure 4: A qualitative evaluation of the 8-bit quantized object detection model on $256\times192$ input images.
• Figure 5: A failed localization scenario due to ambiguity in both geometric and semantic features. The particles, marked as green dots, are divided between two rooms with similar properties. The weighted-average prediction is marked with a red cross.