Neuro-LIFT: A Neuromorphic, LLM-based Interactive Framework for Autonomous Drone FlighT at the Edge
Amogh Joshi, Sourav Sanyal, Kaushik Roy
TL;DR
This work tackles the challenge of enabling intuitive, real-time human control for latency-sensitive autonomous drone navigation. It presents Neuro-LIFT, a modular framework that combines a fine-tuned LLM-based Human Interaction Module with neuromorphic sensing (DVS-based vision) and a physics-aware planner (EV-Planner) to execute high-level instructions as safe, low-latency flight plans. On a constrained indoor platform (Parrot Bebop2), the system demonstrates robust obstacle avoidance and dynamic ring navigation, achieving a maneuvering accuracy of $97.5\%$ ($39/40$ maneuvers) while using safety go/no-go gating and randomized conditions to test resilience. By integrating LLMs with event-based perception and physics-guided planning, Neuro-LIFT offers a practical path toward energy-efficient, interactive edge navigation for agile autonomous drones.
Abstract
The integration of human-intuitive interactions into autonomous systems has been limited. Traditional Natural Language Processing (NLP) systems struggle with context and intent understanding, severely restricting human-robot interaction. Recent advancements in Large Language Models (LLMs) have transformed this dynamic, allowing for intuitive and high-level communication through speech and text, and bridging the gap between human commands and robotic actions. Additionally, autonomous navigation has emerged as a central focus in robotics research, with artificial intelligence (AI) increasingly being leveraged to enhance these systems. However, existing AI-based navigation algorithms face significant challenges in latency-critical tasks where rapid decision-making is critical. Traditional frame-based vision systems, while effective for high-level decision-making, suffer from high energy consumption and latency, limiting their applicability in real-time scenarios. Neuromorphic vision systems, combining event-based cameras and spiking neural networks (SNNs), offer a promising alternative by enabling energy-efficient, low-latency navigation. Despite their potential, real-world implementations of these systems, particularly on physical platforms such as drones, remain scarce. In this work, we present Neuro-LIFT, a real-time neuromorphic navigation framework implemented on a Parrot Bebop2 quadrotor. Leveraging an LLM for natural language processing, Neuro-LIFT translates human speech into high-level planning commands which are then autonomously executed using event-based neuromorphic vision and physics-driven planning. Our framework demonstrates its capabilities in navigating in a dynamic environment, avoiding obstacles, and adapting to human instructions in real-time.