Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Real-Time Neuromorphic Navigation: Integrating Event-Based Vision and Physics-Driven Planning on a Parrot Bebop2 Quadrotor

Amogh Joshi, Sourav Sanyal, Kaushik Roy

TL;DR

This work tackles real-time, energy-efficient autonomous navigation in indoor environments by leveraging neuromorphic event-based vision and physics-driven planning on a small quadrotor. The authors implement an end-to-end pipeline where a DVS camera feeds a spiking neural network-based object detector and a physics-informed planner (EV-Planner) that outputs control setpoints executed by a PID on the Bebop2. They demonstrate real-time tracking of a moving ring with low latency and analyze energy consumption, reporting a modest discrepancy between simulation and reality due to motor and control non-idealities. The work highlights the practicality of neuromorphic sensing for robust, energy-aware aerial navigation and outlines routes to further optimize sensor integration and energy performance.

Abstract

In autonomous aerial navigation, real-time and energy-efficient obstacle avoidance remains a significant challenge, especially in dynamic and complex indoor environments. This work presents a novel integration of neuromorphic event cameras with physics-driven planning algorithms implemented on a Parrot Bebop2 quadrotor. Neuromorphic event cameras, characterized by their high dynamic range and low latency, offer significant advantages over traditional frame-based systems, particularly in poor lighting conditions or during high-speed maneuvers. We use a DVS camera with a shallow Spiking Neural Network (SNN) for event-based object detection of a moving ring in real-time in an indoor lab. Further, we enhance drone control with physics-guided empirical knowledge inside a neural network training mechanism, to predict energy-efficient flight paths to fly through the moving ring. This integration results in a real-time, low-latency navigation system capable of dynamically responding to environmental changes while minimizing energy consumption. We detail our hardware setup, control loop, and modifications necessary for real-world applications, including the challenges of sensor integration without burdening the flight capabilities. Experimental results demonstrate the effectiveness of our approach in achieving robust, collision-free, and energy-efficient flight paths, showcasing the potential of neuromorphic vision and physics-driven planning in enhancing autonomous navigation systems.

Real-Time Neuromorphic Navigation: Integrating Event-Based Vision and Physics-Driven Planning on a Parrot Bebop2 Quadrotor

TL;DR

This work tackles real-time, energy-efficient autonomous navigation in indoor environments by leveraging neuromorphic event-based vision and physics-driven planning on a small quadrotor. The authors implement an end-to-end pipeline where a DVS camera feeds a spiking neural network-based object detector and a physics-informed planner (EV-Planner) that outputs control setpoints executed by a PID on the Bebop2. They demonstrate real-time tracking of a moving ring with low latency and analyze energy consumption, reporting a modest discrepancy between simulation and reality due to motor and control non-idealities. The work highlights the practicality of neuromorphic sensing for robust, energy-aware aerial navigation and outlines routes to further optimize sensor integration and energy performance.

Abstract

In autonomous aerial navigation, real-time and energy-efficient obstacle avoidance remains a significant challenge, especially in dynamic and complex indoor environments. This work presents a novel integration of neuromorphic event cameras with physics-driven planning algorithms implemented on a Parrot Bebop2 quadrotor. Neuromorphic event cameras, characterized by their high dynamic range and low latency, offer significant advantages over traditional frame-based systems, particularly in poor lighting conditions or during high-speed maneuvers. We use a DVS camera with a shallow Spiking Neural Network (SNN) for event-based object detection of a moving ring in real-time in an indoor lab. Further, we enhance drone control with physics-guided empirical knowledge inside a neural network training mechanism, to predict energy-efficient flight paths to fly through the moving ring. This integration results in a real-time, low-latency navigation system capable of dynamically responding to environmental changes while minimizing energy consumption. We detail our hardware setup, control loop, and modifications necessary for real-world applications, including the challenges of sensor integration without burdening the flight capabilities. Experimental results demonstrate the effectiveness of our approach in achieving robust, collision-free, and energy-efficient flight paths, showcasing the potential of neuromorphic vision and physics-driven planning in enhancing autonomous navigation systems.
Paper Structure (5 sections, 2 figures)

This paper contains 5 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. System Overview and Hardware Implementation
  4. Results and Discussion
  5. Conclusion

Figures (2)

  • Figure 1: High-level overview of our real-time neuromorphic navigation setup. Drone and DVS sensor poses are taken from the Optitrack motion capture system consisting of 12 IR cameras. The ring is tracked using the DVS Sensor. The algorithm is executed on an Off-Board NVIDIA Jetson Nano edge processor to publish control commands to the Parrot Bebop2 over a private WiFi network.
  • Figure 2: Flight Trajectories of EV-Planner (simulation) and EV-PID (real-world implementation) while flying through the moving ring without collision. Deviations from the EV-Planner's planned path due to controller artifacts and motor non-idealities are clearly visible which is also reflected in the flight energy values with $\sim15\%$ higher actuation energy observed for EV-PID compared to EV-Planner.