Deploying Foundation Model-Enabled Air and Ground Robots in the Field: Challenges and Opportunities

TL;DR

This work addresses deploying foundation-model enabled robots in large, unstructured field environments by introducing SPINE, an LLM-driven autonomy framework that couples plan generation with plan validation in a closed loop. It demonstrates two planning paradigms: server-based LLMs (e.g., GPT-4o) for field-wide autonomy and onboard distillation of lightweight LMs to enable UAV planning without persistent connectivity. Field experiments with UGVs and UAVs show kilometer-scale missions, revealing strengths in online validation and modular autonomy, while highlighting challenges in communication reliability and long-horizon reasoning on edge devices. The paper also explores air-ground teaming via hierarchical semantic graphs and discusses open challenges, including edge LMs, visual foundation models, and the need for standardized outdoor evaluation protocols.

Abstract

The integration of foundation models (FMs) into robotics has enabled robots to understand natural language and reason about the semantics in their environments. However, existing FM-enabled robots primary operate in closed-world settings, where the robot is given a full prior map or has a full view of its workspace. This paper addresses the deployment of FM-enabled robots in the field, where missions often require a robot to operate in large-scale and unstructured environments. To effectively accomplish these missions, robots must actively explore their environments, navigate obstacle-cluttered terrain, handle unexpected sensor inputs, and operate with compute constraints. We discuss recent deployments of SPINE, our LLM-enabled autonomy framework, in field robotic settings. To the best of our knowledge, we present the first demonstration of large-scale LLM-enabled robot planning in unstructured environments with several kilometers of missions. SPINE is agnostic to a particular LLM, which allows us to distill small language models capable of running onboard size, weight and power (SWaP) limited platforms. Via preliminary model distillation work, we then present the first language-driven UAV planner using on-device language models. We conclude our paper by proposing several promising directions for future research.

Figures (8)

• Figure 1: Example missions. Top: The UGV is given a language-specified mission in a partially-known environment, and it performs several steps of reasoning, planning, and active mapping in order to fulfill the user's request. Bottom: Deployment of a distilled language model running onboard a UAV for planning.
• Figure 2: UGV/UAV autonomy overview. Given a language-specified mission, SPINE uses an LLM to generate an appropriate task sequence, which is validated online for physical safety and realizability. These subtasks are executed by a controller capable of performing atomic behaviors such as point navigation or exploration. The planner can also configures a mapping framework, from which it receives real-time updates. Adapted from ravichandran_spine.
• Figure 3: Overview of the robotic platforms and sensors used in our experiments. Top: high-altitude Falcon 4 UAV. Bottom: Clearpath Jackal. Figure from cladera2025tfr.
• Figure 4: We consider three experimental environments: Top: an urban office park. Middle: a semi-urban environment. bottom: a rural environment (figure adapted from cladera2025tfr).
• Figure 5: Importance of validation. Even minimal feedback significantly improved performance of LLMs. Figure from ravichandran_spine).