Air-Ground Collaboration for Language-Specified Missions in Unknown Environments

TL;DR

This work tackles the challenge of executing language-specified missions with heterogeneous air-ground robot teams in unknown environments by combining a decentralized semantic-metric mapping framework with an LLM-enabled planner (SPINE) and opportunistic communications (MOCHA). The UAV incrementally builds and shares a sparse semantic-topological map, which the UGV uses alongside an onboard LLM-driven planner to reason over subtasks and adapt to changing specifications. Key contributions include a first air-ground system for language-directed missions in unknown environments, a compact shared semantic map for multi-robot coordination, and kilometer-scale demonstrations across urban and rural settings with seven NL specifications. The results show that language-guided air-ground collaboration can outperform UGV-alone approaches when priors are unavailable, while highlighting areas for improvement in traversability estimation, odometry robustness, and onboard language processing for fully autonomous operation.

Abstract

As autonomous robotic systems become increasingly mature, users will want to specify missions at the level of intent rather than in low-level detail. Language is an expressive and intuitive medium for such mission specification. However, realizing language-guided robotic teams requires overcoming significant technical hurdles. Interpreting and realizing language-specified missions requires advanced semantic reasoning. Successful heterogeneous robots must effectively coordinate actions and share information across varying viewpoints. Additionally, communication between robots is typically intermittent, necessitating robust strategies that leverage communication opportunities to maintain coordination and achieve mission objectives. In this work, we present a first-of-its-kind system where an unmanned aerial vehicle (UAV) and an unmanned ground vehicle (UGV) are able to collaboratively accomplish missions specified in natural language while reacting to changes in specification on the fly. We leverage a Large Language Model (LLM)-enabled planner to reason over semantic-metric maps that are built online and opportunistically shared between an aerial and a ground robot. We consider task-driven navigation in urban and rural areas. Our system must infer mission-relevant semantics and actively acquire information via semantic mapping. In both ground and air-ground teaming experiments, we demonstrate our system on seven different natural-language specifications at up to kilometer-scale navigation.

Figures (24)

• Figure 1: Example mission specification. The and start their mission in the same location. The can task the with a list of labels to look for. Once it receives a graph, the can plan on the map, looking for objects of interest. Satellite image in the background is only provided for representation purposes.
• Figure 2: System overview. Green lines represent information that is transmitted to and from MOCHA and thus are communicated to other robots in the system.
• Figure 3: UAV Mapping system with dynamic input from the UGV. (1) A user defines a task which is passed to the SPINE planner. (2) The SPINE Planner assigns the UAV semantic classes to look for. (3) The UAV mapper parses the detections from Grounded SAM2 to produce a map. (4) The map is shared with the UGVs.
• Figure 4: An example of the generated semantic-metric map from the aerial robot is shown at different mapping iterations overlayed on Mapbox Mapbox satellite imagery. The estimated traversable graph is shown in blue and the semantics (in this example orange barrel) is shown in green. Left: Mapping iteration=4, Center: Mapping iteration=12; Right: Mapping iteration=25.
• Figure 5: Example API use during mission. (1) user specifies a mission. (2) SPINE infers relevant semantic categories and configures perception labels. (3) SPINE then reasons about exploration targets. (4) Perception finds a car and notifies planner. (5) Planner informs user.