Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

VLM-RRT: Vision Language Model Guided RRT Search for Autonomous UAV Navigation

Jianlin Ye, Savvas Papaioannou, Panayiotis Kolios

TL;DR

VLM-RRT introduces a hybrid path-planning framework that couples Vision-Language Models with RRT to bias sampling toward regions more likely to yield feasible, efficient UAV paths. By analyzing environmental snapshots with a VLM and using prompting strategies to extract directional guidance, the method accelerates convergence and improves path quality relative to baseline RRT and, in some cases, RRT*. The approach demonstrates robustness to dynamic goal changes and shows that carefully tuned VLM guidance (via prompts and sampling probability) can significantly enhance autonomous navigation in hazardous environments. This perception-guided sampling paradigm has practical potential for rapid, reliable UAV planning in disaster response and similar high-stakes scenarios, with future work aimed at deeper LLM integration and real-world validation.

Abstract

Path planning is a fundamental capability of autonomous Unmanned Aerial Vehicles (UAVs), enabling them to efficiently navigate toward a target region or explore complex environments while avoiding obstacles. Traditional pathplanning methods, such as Rapidly-exploring Random Trees (RRT), have proven effective but often encounter significant challenges. These include high search space complexity, suboptimal path quality, and slow convergence, issues that are particularly problematic in high-stakes applications like disaster response, where rapid and efficient planning is critical. To address these limitations and enhance path-planning efficiency, we propose Vision Language Model RRT (VLM-RRT), a hybrid approach that integrates the pattern recognition capabilities of Vision Language Models (VLMs) with the path-planning strengths of RRT. By leveraging VLMs to provide initial directional guidance based on environmental snapshots, our method biases sampling toward regions more likely to contain feasible paths, significantly improving sampling efficiency and path quality. Extensive quantitative and qualitative experiments with various state-of-the-art VLMs demonstrate the effectiveness of this proposed approach.

VLM-RRT: Vision Language Model Guided RRT Search for Autonomous UAV Navigation

TL;DR

VLM-RRT introduces a hybrid path-planning framework that couples Vision-Language Models with RRT to bias sampling toward regions more likely to yield feasible, efficient UAV paths. By analyzing environmental snapshots with a VLM and using prompting strategies to extract directional guidance, the method accelerates convergence and improves path quality relative to baseline RRT and, in some cases, RRT*. The approach demonstrates robustness to dynamic goal changes and shows that carefully tuned VLM guidance (via prompts and sampling probability) can significantly enhance autonomous navigation in hazardous environments. This perception-guided sampling paradigm has practical potential for rapid, reliable UAV planning in disaster response and similar high-stakes scenarios, with future work aimed at deeper LLM integration and real-world validation.

Abstract

Path planning is a fundamental capability of autonomous Unmanned Aerial Vehicles (UAVs), enabling them to efficiently navigate toward a target region or explore complex environments while avoiding obstacles. Traditional pathplanning methods, such as Rapidly-exploring Random Trees (RRT), have proven effective but often encounter significant challenges. These include high search space complexity, suboptimal path quality, and slow convergence, issues that are particularly problematic in high-stakes applications like disaster response, where rapid and efficient planning is critical. To address these limitations and enhance path-planning efficiency, we propose Vision Language Model RRT (VLM-RRT), a hybrid approach that integrates the pattern recognition capabilities of Vision Language Models (VLMs) with the path-planning strengths of RRT. By leveraging VLMs to provide initial directional guidance based on environmental snapshots, our method biases sampling toward regions more likely to contain feasible paths, significantly improving sampling efficiency and path quality. Extensive quantitative and qualitative experiments with various state-of-the-art VLMs demonstrate the effectiveness of this proposed approach.

Paper Structure

This paper contains 13 sections, 3 equations, 4 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. UAV Dynamical Model
  5. Environment Model
  6. Problem Formulation
  7. Proposed Approach
  8. VLM-RRT
  9. Prompt Engineering
  10. Evaluation
  11. Simulation Setup
  12. Results
  13. Conclusion and Future Work

Figures (4)

  • Figure 1: Our basic system consists of two types of prompts, task descriptions and basic inputs. We match a snapshot of the current environment with the task instructions, incorporating the current navigation state and history into the prompt to activate the agent's global dynamic exploration capability.
  • Figure 2: Comparison of different prompt engineering techniques for navigation decision-making.
  • Figure 3: Illustrative example of the path-planning behavior obtained with: (a) RRT, (b) RRT* and (c) VLM-RRT.
  • Figure 4: Illustration of the VLM-RRT algorithm navigating toward dynamic goals in a 2D environment. The scenario involves three goal relocations. Red point is the starting position, green point is the goal position which changes over time, and the blue point is a leaf node. The yellow sector is the VLM-informed sampling region.