Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UAV-VLN: End-to-End Vision Language guided Navigation for UAVs

Pranav Saxena, Nishant Raghuvanshi, Neena Goveas

TL;DR

The paper addresses end-to-end vision-language guided navigation for UAVs by introducing UAV-VLN, an architecture that grounds natural language instructions in real-time visual perception to generate executable 3D flight plans. It combines a domain-tuned LLM (TinyLlama-1.1B) for instruction decomposition, Grounding DINO for open-vocabulary grounding, and a ROS 2-based automated task planner to translate sub-goals into low-level UAV commands. A novel UAV navigation instruction dataset (~1,000 prompts) and comprehensive experiments in indoor/outdoor scenes demonstrate improved instruction-following accuracy and trajectory efficiency over baselines, highlighting the potential of LLM-driven vision-language interfaces for safe, generalizable UAV autonomy with minimal task-specific supervision. The work establishes a practical, end-to-end framework that can adapt to diverse, unstructured environments and paves the way for future enhancements in global reasoning via navigation history and semantic mapping.

Abstract

A core challenge in AI-guided autonomy is enabling agents to navigate realistically and effectively in previously unseen environments based on natural language commands. We propose UAV-VLN, a novel end-to-end Vision-Language Navigation (VLN) framework for Unmanned Aerial Vehicles (UAVs) that seamlessly integrates Large Language Models (LLMs) with visual perception to facilitate human-interactive navigation. Our system interprets free-form natural language instructions, grounds them into visual observations, and plans feasible aerial trajectories in diverse environments. UAV-VLN leverages the common-sense reasoning capabilities of LLMs to parse high-level semantic goals, while a vision model detects and localizes semantically relevant objects in the environment. By fusing these modalities, the UAV can reason about spatial relationships, disambiguate references in human instructions, and plan context-aware behaviors with minimal task-specific supervision. To ensure robust and interpretable decision-making, the framework includes a cross-modal grounding mechanism that aligns linguistic intent with visual context. We evaluate UAV-VLN across diverse indoor and outdoor navigation scenarios, demonstrating its ability to generalize to novel instructions and environments with minimal task-specific training. Our results show significant improvements in instruction-following accuracy and trajectory efficiency, highlighting the potential of LLM-driven vision-language interfaces for safe, intuitive, and generalizable UAV autonomy.

UAV-VLN: End-to-End Vision Language guided Navigation for UAVs

TL;DR

The paper addresses end-to-end vision-language guided navigation for UAVs by introducing UAV-VLN, an architecture that grounds natural language instructions in real-time visual perception to generate executable 3D flight plans. It combines a domain-tuned LLM (TinyLlama-1.1B) for instruction decomposition, Grounding DINO for open-vocabulary grounding, and a ROS 2-based automated task planner to translate sub-goals into low-level UAV commands. A novel UAV navigation instruction dataset (~1,000 prompts) and comprehensive experiments in indoor/outdoor scenes demonstrate improved instruction-following accuracy and trajectory efficiency over baselines, highlighting the potential of LLM-driven vision-language interfaces for safe, generalizable UAV autonomy with minimal task-specific supervision. The work establishes a practical, end-to-end framework that can adapt to diverse, unstructured environments and paves the way for future enhancements in global reasoning via navigation history and semantic mapping.

Abstract

A core challenge in AI-guided autonomy is enabling agents to navigate realistically and effectively in previously unseen environments based on natural language commands. We propose UAV-VLN, a novel end-to-end Vision-Language Navigation (VLN) framework for Unmanned Aerial Vehicles (UAVs) that seamlessly integrates Large Language Models (LLMs) with visual perception to facilitate human-interactive navigation. Our system interprets free-form natural language instructions, grounds them into visual observations, and plans feasible aerial trajectories in diverse environments. UAV-VLN leverages the common-sense reasoning capabilities of LLMs to parse high-level semantic goals, while a vision model detects and localizes semantically relevant objects in the environment. By fusing these modalities, the UAV can reason about spatial relationships, disambiguate references in human instructions, and plan context-aware behaviors with minimal task-specific supervision. To ensure robust and interpretable decision-making, the framework includes a cross-modal grounding mechanism that aligns linguistic intent with visual context. We evaluate UAV-VLN across diverse indoor and outdoor navigation scenarios, demonstrating its ability to generalize to novel instructions and environments with minimal task-specific training. Our results show significant improvements in instruction-following accuracy and trajectory efficiency, highlighting the potential of LLM-driven vision-language interfaces for safe, intuitive, and generalizable UAV autonomy.
Paper Structure (17 sections, 3 figures, 2 tables)

This paper contains 17 sections, 3 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. Related Work
  3. Vision-and-Language Navigation
  4. Aerial Navigation
  5. Foundation Models for Embodied Navigation
  6. OUR APPROACH
  7. Problem Definition
  8. Natural Language Prompting
  9. Automated Task Planner
  10. Visual Input
  11. Termination
  12. EXPERIMENTS
  13. Procedure
  14. Evaluation
  15. Baselines
  16. ...and 2 more sections

Figures (3)

  • Figure 1: An example of VLN episode. A human user queries a LLM with the task prompt to generate sub-goals. An automated task planner generates a task plan with respect to the sub-goals for the drone to execute. The drone also uses the visual input to carry out the task plan.
  • Figure 2: The system architecture of UAV-VLN with four sequential stages: (1) Natural language prompt and action space are provided as inputs; (2) a fine-tuned LLM performs semantic decomposition of the instruction into structured sub-goals; (3) an automated task planner maps each sub-goal to executable low-level UAV actions considering environmental context; (4) the generated sub-plans are synthesized into a coherent final mission plan ensuring instruction consistency, safety, and robustness.
  • Figure 3: An example of an input image and natural language prompt used in UAV-VLN. (i) The prompt is processed by TinyLLaMA to generate a high-level action sequence. (ii) An automated task planner identifies the relevant target objects in the sequence. (iii) The input image and identified objects are passed to Grounding-DINO to obtain bounding boxes. (iv) The action sequence is then mapped to low-level drone commands for execution.