SPINE: Online Semantic Planning for Missions with Incomplete Natural Language Specifications in Unstructured Environments

TL;DR

SPINE introduces an online semantic planner capable of executing missions described in incomplete natural language within partially known, unstructured environments. It combines an LLM-based plan generator with a separate plan validator and a topological semantic mapper to infer and realize subtasks in a receding-horizon loop, updating the map as new observations arrive. The approach obviates the need for a full prior map, enabling efficient navigation, active mapping, and user interaction while maintaining safety through validation. Empirical results in both large-scale simulations and real-robot experiments show SPINE achieving high mission success with significantly reduced time and traversal distance compared to baselines that rely on fully specified maps or explicit tasking, with online validation proving crucial as environmental certainty declines.

Abstract

As robots become increasingly capable, users will want to describe high-level missions and have robots infer the relevant details. Because pre-built maps are difficult to obtain in many realistic settings, accomplishing such missions will require the robot to map and plan online. While many semantic planning methods operate online, they are typically designed for well specified missions such as object search or exploration. Recently, Large Language Models (LLMs) have demonstrated powerful contextual reasoning abilities over a range of robotic tasks described in natural language. However, existing LLM-enabled planners typically do not consider online planning or complex missions; rather, relevant subtasks and semantics are provided by a pre-built map or a user. We address these limitations via SPINE, an online planner for missions with incomplete mission specifications provided in natural language. The planner uses an LLM to reason about subtasks implied by the mission specification and then realizes these subtasks in a receding horizon framework. Tasks are automatically validated for safety and refined online with new map observations. We evaluate SPINE in simulation and real-world settings with missions that require multiple steps of semantic reasoning and exploration in cluttered outdoor environments of over 20,000m$^2$. Compared to baselines that use existing LLM-enabled planning approaches, our method is over twice as efficient in terms of time and distance, requires less user interactions, and does not require a full map. Additional resources are provided at https://zacravichandran.github.io/SPINE.

Figures (21)

• Figure 1: (A) SPINE takes as input a mission with incomplete specifications and prior map. (B) SPINE then reasons about the goals and semantics required to achieve the mission. (C) SPINE's plan generator uses an LLM to generate subtasks, while its validation module ensures those subtasks are realizable. (D) SPINE then actively explores and reasons over acquired information in order to complete its mission.
• Figure 2: SPINE architecture. A user provides SPINE with a mission specification. SPINE plans via behaviors for user interaction, active mapping, and robot control. SPINE's plan generator infers a task sequence which is validated online for correctness and feasibility; if necessary, feedback and corrections are provided in real-time. Actions are sent to the appropriate module, and the planner refines its plan as new information is acquired.
• Figure 3: Online validation enables exploration. SPINE's plan generator may produce exploration goal outside robot's obstacle map, which may not be reachable. Spatial validation iteratively finds best reachable fit, and the robot navigates to that point. Procedure terminates once robot reaches its goal
• Figure 4: Experimental platform, 3D view of environment, and example prior and corresponding task used for real-world experiments. The prior map is derived from outdated satellite imagery or obstructed due to trees and other coverings. The prior map is thus incomplete and partially incorrect, which requires the planner to reason about information acquired online.
• Figure 5: Given a mission and prior map, SPINE must (2) explore and (3, 4) visit and inspect communication infrastructure. SPINE then forms an appropriate inspection query for the mapper's vision language model (VLM), and it uses the acquired information to solve the mission.