Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DroneVLA: VLA based Aerial Manipulation

Fawad Mehboob, Monijesu James, Amir Habel, Jeffrin Sam, Miguel Altamirano Cabrera, Dzmitry Tsetserukou

TL;DR

DroneVLA tackles intuitive, safe aerial manipulation by enabling natural-language prompts to command a drone equipped with a gripper. It combines a Vision-Language-Action module with Grounding DINO-based perception, a human-centric handover controller guided by MediaPipe, and a grid-based A* planner that accounts for the human, effectively decoupling semantic reasoning from flight control. Real-world indoor experiments report localization errors of max $0.164$ m, mean $0.070$ m, and RMSE $0.084$ m, with a safe stand-off of about $1$ m during handover, demonstrating feasibility of VLA-driven aerial manipulation. The work provides a modular, data-efficient pathway toward embodied aerial intelligence and outlines steps toward full $5$-DOF closed-loop control and industrial deployment.

Abstract

As aerial platforms evolve from passive observers to active manipulators, the challenge shifts toward designing intuitive interfaces that allow non-expert users to command these systems naturally. This work introduces a novel concept of autonomous aerial manipulation system capable of interpreting high-level natural language commands to retrieve objects and deliver them to a human user. The system is intended to integrate a MediaPipe based on Grounding DINO and a Vision-Language-Action (VLA) model with a custom-built drone equipped with a 1-DOF gripper and an Intel RealSense RGB-D camera. VLA performs semantic reasoning to interpret the intent of a user prompt and generates a prioritized task queue for grasping of relevant objects in the scene. Grounding DINO and dynamic A* planning algorithm are used to navigate and safely relocate the object. To ensure safe and natural interaction during the handover phase, the system employs a human-centric controller driven by MediaPipe. This module provides real-time human pose estimation, allowing the drone to employ visual servoing to maintain a stable, distinct position directly in front of the user, facilitating a comfortable handover. We demonstrate the system's efficacy through real-world experiments for localization and navigation, which resulted in a 0.164m, 0.070m, and 0.084m of max, mean euclidean, and root-mean squared errors, respectively, highlighting the feasibility of VLA for aerial manipulation operations.

DroneVLA: VLA based Aerial Manipulation

TL;DR

DroneVLA tackles intuitive, safe aerial manipulation by enabling natural-language prompts to command a drone equipped with a gripper. It combines a Vision-Language-Action module with Grounding DINO-based perception, a human-centric handover controller guided by MediaPipe, and a grid-based A* planner that accounts for the human, effectively decoupling semantic reasoning from flight control. Real-world indoor experiments report localization errors of max m, mean m, and RMSE m, with a safe stand-off of about m during handover, demonstrating feasibility of VLA-driven aerial manipulation. The work provides a modular, data-efficient pathway toward embodied aerial intelligence and outlines steps toward full -DOF closed-loop control and industrial deployment.

Abstract

As aerial platforms evolve from passive observers to active manipulators, the challenge shifts toward designing intuitive interfaces that allow non-expert users to command these systems naturally. This work introduces a novel concept of autonomous aerial manipulation system capable of interpreting high-level natural language commands to retrieve objects and deliver them to a human user. The system is intended to integrate a MediaPipe based on Grounding DINO and a Vision-Language-Action (VLA) model with a custom-built drone equipped with a 1-DOF gripper and an Intel RealSense RGB-D camera. VLA performs semantic reasoning to interpret the intent of a user prompt and generates a prioritized task queue for grasping of relevant objects in the scene. Grounding DINO and dynamic A* planning algorithm are used to navigate and safely relocate the object. To ensure safe and natural interaction during the handover phase, the system employs a human-centric controller driven by MediaPipe. This module provides real-time human pose estimation, allowing the drone to employ visual servoing to maintain a stable, distinct position directly in front of the user, facilitating a comfortable handover. We demonstrate the system's efficacy through real-world experiments for localization and navigation, which resulted in a 0.164m, 0.070m, and 0.084m of max, mean euclidean, and root-mean squared errors, respectively, highlighting the feasibility of VLA for aerial manipulation operations.
Paper Structure (11 sections, 4 equations, 4 figures)

This paper contains 11 sections, 4 equations, 4 figures.

Table of Contents

  1. Introduction
  2. System Architecture
  3. Hardware Platform
  4. Vision-Language-Action (VLA) Module
  5. Perception Pipeline: Open-Vocabulary Detection and 3D Localization
  6. Human-Centric Handover Controller
  7. Motion planning with human awareness
  8. System Evaluation
  9. Experimental Procedure
  10. Discussion
  11. Conclusion and Future Work

Figures (4)

  • Figure 1: Human Localization and Navigation by drone
  • Figure 2: Overview of the VLA based aerial manipulation and Path Planning Architecture
  • Figure 3: Grounding DINO bounding box object detection and humans in RGB images from drone cameras
  • Figure 4: 2D and 3D views of the human–aware A* motion plan, showing the reference trajectory, planned path, safety margins around the human and table with achieved trajectory in real flight.