Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Sim-to-Real Deep Learning-based Framework for Autonomous Nano-drone Racing

Lorenzo Lamberti, Elia Cereda, Gabriele Abbate, Lorenzo Bellone, Victor Javier Kartsch Morinigo, Michał Barcis, Agata Barcis, Alessandro Giusti, Francesco Conti, Daniele Palossi

TL;DR

This work tackles autonomous nano-drone racing under severe onboard compute constraints by deploying a fully onboard CNN-based obstacle avoidance system trained only on simulated data. A sim-to-real mitigation strategy, aggressive photometric augmentation, and fixed-point deployment enable robust, real-time inference on a GAP8-based platform. The authors propose three navigation policies, including a waypoint-based strategy, and demonstrate their approach through extensive in-sim and in-field experiments, culminating in a first-place finish at the IMAV'22 Nanocopter AI Challenge with 115 m traveled distance and no crashes. The results advance the feasibility of onboard, lightweight perception for ultra-small UAVs and provide a practical foundation for extending onboard autonomy to gate-based navigation in future work.

Abstract

Autonomous drone racing competitions are a proxy to improve unmanned aerial vehicles' perception, planning, and control skills. The recent emergence of autonomous nano-sized drone racing imposes new challenges, as their ~10cm form factor heavily restricts the resources available onboard, including memory, computation, and sensors. This paper describes the methodology and technical implementation of the system winning the first autonomous nano-drone racing international competition: the IMAV 2022 Nanocopter AI Challenge. We developed a fully onboard deep learning approach for visual navigation trained only on simulation images to achieve this goal. Our approach includes a convolutional neural network for obstacle avoidance, a sim-to-real dataset collection procedure, and a navigation policy that we selected, characterized, and adapted through simulation and actual in-field experiments. Our system ranked 1st among seven competing teams at the competition. In our best attempt, we scored 115m of traveled distance in the allotted 5-minute flight, never crashing while dodging static and dynamic obstacles. Sharing our knowledge with the research community, we aim to provide a solid groundwork to foster future development in this field.

A Sim-to-Real Deep Learning-based Framework for Autonomous Nano-drone Racing

TL;DR

This work tackles autonomous nano-drone racing under severe onboard compute constraints by deploying a fully onboard CNN-based obstacle avoidance system trained only on simulated data. A sim-to-real mitigation strategy, aggressive photometric augmentation, and fixed-point deployment enable robust, real-time inference on a GAP8-based platform. The authors propose three navigation policies, including a waypoint-based strategy, and demonstrate their approach through extensive in-sim and in-field experiments, culminating in a first-place finish at the IMAV'22 Nanocopter AI Challenge with 115 m traveled distance and no crashes. The results advance the feasibility of onboard, lightweight perception for ultra-small UAVs and provide a practical foundation for extending onboard autonomy to gate-based navigation in future work.

Abstract

Autonomous drone racing competitions are a proxy to improve unmanned aerial vehicles' perception, planning, and control skills. The recent emergence of autonomous nano-sized drone racing imposes new challenges, as their ~10cm form factor heavily restricts the resources available onboard, including memory, computation, and sensors. This paper describes the methodology and technical implementation of the system winning the first autonomous nano-drone racing international competition: the IMAV 2022 Nanocopter AI Challenge. We developed a fully onboard deep learning approach for visual navigation trained only on simulation images to achieve this goal. Our approach includes a convolutional neural network for obstacle avoidance, a sim-to-real dataset collection procedure, and a navigation policy that we selected, characterized, and adapted through simulation and actual in-field experiments. Our system ranked 1st among seven competing teams at the competition. In our best attempt, we scored 115m of traveled distance in the allotted 5-minute flight, never crashing while dodging static and dynamic obstacles. Sharing our knowledge with the research community, we aim to provide a solid groundwork to foster future development in this field.
Paper Structure (11 sections, 1 equation, 10 figures, 3 tables)

This paper contains 11 sections, 1 equation, 10 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. The IMAV'22 Nanocopter AI Challenge
  4. Navigation policies
  5. CNN training and deployment
  6. Experimental results
  7. In-simulator evaluation
  8. State-of-the-Art comparison
  9. In-field evaluation
  10. The nanocopter AI challenge
  11. Conclusion and future work

Figures (10)

  • Figure 1: Our nano-drone winning the IMAV'22 "Nanocopter AI Challenge."
  • Figure 2: A) the 10$\times$10m competition arena. B) the robotic platform.
  • Figure 3: A) 1024 simulated Monte Carlo trajectory realizations B) minimum safety margin w.r.t. height, C) minimum safety margin w.r.t speed, having fixed the height to 0.5m.
  • Figure 4: Images: A) from the simulator, B) after augmentation, C) Himax camera sample collected in the IMAV arena. The three blue bars in images B-C) represent the three collision probabilities predicted by our network.
  • Figure 5: The drone's state machines mapped on the MCUs available aboard.
  • ...and 5 more figures