Next-Generation LLM for UAV: From Natural Language to Autonomous Flight

TL;DR

NeLV presents an end-to-end framework that maps natural-language instructions to executable UAV trajectories across short-, medium-, and long-range operations using a five-component pipeline and a five-level automation roadmap. The system integrates LLM-based instruction parsing (LLM-as-Parser) with route planning, path planning, control, and monitoring, leveraging multi-objective optimization and PSO for path generation, and airport-aware patterns for take-off and landing. Three case studies illustrate multi-UAV patrol, multi-POI delivery, and multi-hop relocation, using diverse data sources (OSM, Yelp, AirNav) and real weather and airspace data. The roadmap outlines progress from current NLP-driven parsing to fully autonomous LLM-powered autopilots, highlighting data, regulatory, and safety challenges and the need for UAV-specific datasets and simulators to advance domain.

Abstract

With the rapid advancement of Large Language Models (LLMs), their capabilities in various automation domains, particularly Unmanned Aerial Vehicle (UAV) operations, have garnered increasing attention. Current research remains predominantly constrained to small-scale UAV applications, with most studies focusing on isolated components such as path planning for toy drones, while lacking comprehensive investigation of medium- and long-range UAV systems in real-world operational contexts. Larger UAV platforms introduce distinct challenges, including stringent requirements for airport-based take-off and landing procedures, adherence to complex regulatory frameworks, and specialized operational capabilities with elevated mission expectations. This position paper presents the Next-Generation LLM for UAV (NeLV) system -- a comprehensive demonstration and automation roadmap for integrating LLMs into multi-scale UAV operations. The NeLV system processes natural language instructions to orchestrate short-, medium-, and long-range UAV missions through five key technical components: (i) LLM-as-Parser for instruction interpretation, (ii) Route Planner for Points of Interest (POI) determination, (iii) Path Planner for waypoint generation, (iv) Control Platform for executable trajectory implementation, and (v) UAV monitoring. We demonstrate the system's feasibility through three representative use cases spanning different operational scales: multi-UAV patrol, multi-POI delivery, and multi-hop relocation. Beyond the current implementation, we establish a five-level automation taxonomy that charts the evolution from current LLM-as-Parser capabilities (Level 1) to fully autonomous LLM-as-Autopilot systems (Level 5), identifying technical prerequisites and research challenges at each stage.

Figures (7)

• Figure 1: Overview of the NeLV System.
• Figure 2: Operational Framework and User Interface of the NeLV System. Left – Chat Box: The pilot interacts with the LLM through natural-language instructions. After receiving an initial flight plan, the pilot performs route and path planning with visualization support. Top – Control Platform mission_control: Once the path is uploaded, the pilot can further refine the mission by manually adjusting waypoints. Right – UAV Monitor: The safety pilot continuously monitors UAVs' status, position, and flight behavior during operation and can make real-time adjustments as needed.
• Figure 3: Standard take-off circuit pattern configured in a kidney-shaped trajectory to enable the UAV to maintain controlled loitering within the traffic pattern until air traffic control (ATC) clearance is received. The purple waypoint (0) indicates the initial take-off position and white waypoints represent intermediate navigation points along the flight path. The blue connecting lines illustrate the planned trajectory segments between consecutive waypoints, with the directional arrows indicating the prescribed flight direction and sequence. Waypoint 5 is strategically positioned to prevent the aircraft from overflying the active runway, ensuring compliance with aviation safety protocols and airport operational procedures.
• Figure 4: Use Case 1: Short-Range Multi-UAV Patrol (Sec. \ref{['Sec. Use Case 1']}).
• Figure 5: Use Case 2: Medium-Range Multi-POI Delivery (Sec. \ref{['Sec. Use Case 2']}).