LLM-Land: Large Language Models for Context-Aware Drone Landing

TL;DR

This work addresses safe autonomous drone landing in dynamic, unstructured environments by fusing vision-language perception with a retrieval-augmented LLM that reasoned about scene semantics and safety buffers, integrated with a real-time MPC planner. The LLM-Land framework uses a VLE (BLIP) to generate captions, a lightweight LLM with RAG to output context-aware constraints, and an MPC-based controller to replan trajectories while avoiding dynamic obstacles. Experiments in ROS-Gazebo show substantial improvements in landing success and safety over baselines, and constrained-hardware benchmarks demonstrate feasibility for edge deployment. Overall, the approach bridges high-level semantic understanding with low-level motion control to enable robust, context-aware UAV landings in real-world clutter.

Abstract

Autonomous landing is essential for drones deployed in emergency deliveries, post-disaster response, and other large-scale missions. By enabling self-docking on charging platforms, it facilitates continuous operation and significantly extends mission endurance. However, traditional approaches often fall short in dynamic, unstructured environments due to limited semantic awareness and reliance on fixed, context-insensitive safety margins. To address these limitations, we propose a hybrid framework that integrates large language model (LLMs) with model predictive control (MPC). Our approach begins with a vision-language encoder (VLE) (e.g., BLIP), which transforms real-time images into concise textual scene descriptions. These descriptions are processed by a lightweight LLM (e.g., Qwen 2.5 1.5B or LLaMA 3.2 1B) equipped with retrieval-augmented generation (RAG) to classify scene elements and infer context-aware safety buffers, such as 3 meters for pedestrians and 5 meters for vehicles. The resulting semantic flags and unsafe regions are then fed into an MPC module, enabling real-time trajectory replanning that avoids collisions while maintaining high landing precision. We validate our framework in the ROS-Gazebo simulator, where it consistently outperforms conventional vision-based MPC baselines. Our results show a significant reduction in near-miss incidents with dynamic obstacles, while preserving accurate landings in cluttered environments.

Figures (5)

• Figure 1: Overview of the LLM-Land framework for semantics-driven safe landing. (A) The Perception System receives raw images from the onboard camera and converts them into scene captions using a vision–language pretrained encoder model (e.g., BLIP). An LLM augmented with Retrieval-Augmented Generation (RAG) analyzes each caption to detect dynamic obstacles and output the required unsafe region, emitting a structured JSON specification. (B) The Action System ingests the JSON input and collaborates with a local MPC-based planner to generate actions for a safe landing. Top-right illustration shows the landing action sequence. Bottom panel detailed semantic reasoning process.
• Figure 2: Experiments were conducted using the Gazebo simulator, which provides realistic physics and sensor modeling, including support for both depth and RGB-D cameras. We evaluated LLM-Land across three representative scenarios: (a) Open Field: a large, flat area with minimal static obstacles; (b) Urban / Town: a built-up environment featuring buildings, streets, and parked vehicles; (c) Grassland / Forest Edge: a semi-natural setting with outdoor structures, trees, and underbrush.
• Figure 3: Illustration of the LLM-land pipeline from visual perception to semantic reasoning and trajectory updates.
• Figure 4: Indoor flight test with the custom-built UAV. The UAV detected a manikin near its path, and the LLM-land module classified it as a dynamic obstacle that needs to be avoided actively. The UAV adjusted its trajectory to fly over the manikin and completed a safe landing.
• Figure 5: Lightweight UAV platform based on VOXL and PX4.