Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Universal Large Language Model -- Drone Command and Control Interface

Javier N. Ramos-Silva, Peter J. Burke

TL;DR

This work addresses the barrier of connecting large language models to drone control by introducing a universal Model Context Protocol (MCP) that acts as an intermediary between LLMs and MAVLink-compatible drones. The authors build a cloud-based DroneServer that bridges LLMs (via MCP) to drones through high-level MavSDK commands, while exposing a curated set of tools and maintaining safety through abstraction from low-level Mavlink details. The paper demonstrates real-world drone flights and extensive SITL simulations, including integration with external MCP servers such as Google Maps for up-to-date navigation, illustrating the feasibility of scalable, LLM-driven autonomous missions. The results suggest a pathway to broad adoption of physical AI in drones, enabling multi-LLM and multi-drone ecosystems with open-source tooling and standard interfaces for rapid deployment and experimentation.

Abstract

The use of artificial intelligence (AI) for drone control can have a transformative impact on drone capabilities, especially when real world information can be integrated with drone sensing, command, and control, part of a growing field of physical AI. Large language models (LLMs) can be advantageous if trained at scale on general knowledge, but especially and in particular when the training data includes information such as detailed map geography topology of the entire planet, as well as the ability to access real time situational data such as weather. However, challenges remain in the interface between drones and LLMs in general, with each application requiring a tedious, labor intensive effort to connect the LLM trained knowledge to drone command and control. Here, we solve that problem, using an interface strategy that is LLM agnostic and drone agnostic, providing the first universal, versatile, comprehensive and easy to use drone control interface. We do this using the new model context protocol (MCP) standard, an open standard that provides a universal way for AI systems to access external data, tools, and services. We develop and deploy a cloud based Linux machine hosting an MCP server that supports the Mavlink protocol, an ubiquitous drone control language used almost universally by millions of drones including Ardupilot and PX4 framework.We demonstrate flight control of a real unmanned aerial vehicle. In further testing, we demonstrate extensive flight planning and control capability in a simulated drone, integrated with a Google Maps MCP server for up to date, real time navigation information. This demonstrates a universal approach to integration of LLMs with drone command and control, a paradigm that leverages and exploits virtually all of modern AI industry with drone technology in an easy to use interface that translates natural language to drone control.

A Universal Large Language Model -- Drone Command and Control Interface

TL;DR

This work addresses the barrier of connecting large language models to drone control by introducing a universal Model Context Protocol (MCP) that acts as an intermediary between LLMs and MAVLink-compatible drones. The authors build a cloud-based DroneServer that bridges LLMs (via MCP) to drones through high-level MavSDK commands, while exposing a curated set of tools and maintaining safety through abstraction from low-level Mavlink details. The paper demonstrates real-world drone flights and extensive SITL simulations, including integration with external MCP servers such as Google Maps for up-to-date navigation, illustrating the feasibility of scalable, LLM-driven autonomous missions. The results suggest a pathway to broad adoption of physical AI in drones, enabling multi-LLM and multi-drone ecosystems with open-source tooling and standard interfaces for rapid deployment and experimentation.

Abstract

The use of artificial intelligence (AI) for drone control can have a transformative impact on drone capabilities, especially when real world information can be integrated with drone sensing, command, and control, part of a growing field of physical AI. Large language models (LLMs) can be advantageous if trained at scale on general knowledge, but especially and in particular when the training data includes information such as detailed map geography topology of the entire planet, as well as the ability to access real time situational data such as weather. However, challenges remain in the interface between drones and LLMs in general, with each application requiring a tedious, labor intensive effort to connect the LLM trained knowledge to drone command and control. Here, we solve that problem, using an interface strategy that is LLM agnostic and drone agnostic, providing the first universal, versatile, comprehensive and easy to use drone control interface. We do this using the new model context protocol (MCP) standard, an open standard that provides a universal way for AI systems to access external data, tools, and services. We develop and deploy a cloud based Linux machine hosting an MCP server that supports the Mavlink protocol, an ubiquitous drone control language used almost universally by millions of drones including Ardupilot and PX4 framework.We demonstrate flight control of a real unmanned aerial vehicle. In further testing, we demonstrate extensive flight planning and control capability in a simulated drone, integrated with a Google Maps MCP server for up to date, real time navigation information. This demonstrates a universal approach to integration of LLMs with drone command and control, a paradigm that leverages and exploits virtually all of modern AI industry with drone technology in an easy to use interface that translates natural language to drone control.
Paper Structure (42 sections, 8 figures, 2 tables)

This paper contains 42 sections, 8 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Architecture
  3. Concept of operations
  4. Mavlink protocol
  5. Model context protocol: From manual tools to universal standard LLM interface
  6. Tools: Tedious and manual agentic AI programming
  7. Model Context Protocol
  8. Development method
  9. Server architecture
  10. Internals
  11. LLL-MCP server link
  12. Drone to MCP server link
  13. Low level Mavlink and TCP/IP are abstracted from the LLM
  14. MavSDK is a higher level set of commands used for this work
  15. Beyond one tool per MavSDK method
  16. ...and 27 more sections

Figures (8)

  • Figure 1: Architecture. Any LLM that supports the MCP standard can connect to the MCP server, which in turn provides a low level interface with a drone using Mavlink protocol.
  • Figure 2: Concept of operations. An LLM has access to multiple services, tools, and MCP servers. One of them is this drone control server, but the LLM can access any of thousands of other MCP servers. An example mission is shown integrating both a map (Google maps) and drone control server.
  • Figure 3: Model context protocol (MCP) standard. The MCP server exposes resources, prompts, and tools to the LLM. The LLM does not need to know the details of the implementation of these, and uses them based on the context of the prompt.
  • Figure 4: Tech stack. The tech stack of the MCP server developed in this work. The drone communicates over TCP/IP using Mavlink protocol, while the LLM communications over HTTP using the MCP protocol. The server contains custom code to coordinate all the interactions to provide seemless integration between the LLM and the drone.
  • Figure 5: Picture of drone used in this work. A LIDAR and optical flow sensor is used for GPS-denied flights, for example in the drone cage.
  • ...and 3 more figures