Deploying and Evaluating LLMs to Program Service Mobile Robots

TL;DR

The paper addresses translating natural language tasks into executable robot programs for service mobile robots using an LLM-enabled pipeline. It introduces CodeBotler, a robot-agnostic framework with a Python-embedded DSL to ground tasks in reusable robot primitives and a lightweight deployment via ROS Actionlib, and RoboEval, a symbolic-simulator plus RTL-based benchmark that rigorously tests temporal correctness across multiple initial world states. The study evaluates five state-of-the-art LLMs, characterizes failure modes with a long-tail distribution, and demonstrates that a rejection sampling strategy can reduce execution-time errors, though it cannot fully resolve all failures. It highlights the need for improved grounding and abstraction matching to bridge natural-language intents and precise robotic actions, guiding future work toward more robust, adaptable robot-program generation.

Abstract

Recent advancements in large language models (LLMs) have spurred interest in using them for generating robot programs from natural language, with promising initial results. We investigate the use of LLMs to generate programs for service mobile robots leveraging mobility, perception, and human interaction skills, and where accurate sequencing and ordering of actions is crucial for success. We contribute CodeBotler, an open-source robot-agnostic tool to program service mobile robots from natural language, and RoboEval, a benchmark for evaluating LLMs' capabilities of generating programs to complete service robot tasks. CodeBotler performs program generation via few-shot prompting of LLMs with an embedded domain-specific language (eDSL) in Python, and leverages skill abstractions to deploy generated programs on any general-purpose mobile robot. RoboEval evaluates the correctness of generated programs by checking execution traces starting with multiple initial states, and checking whether the traces satisfy temporal logic properties that encode correctness for each task. RoboEval also includes multiple prompts per task to test for the robustness of program generation. We evaluate several popular state-of-the-art LLMs with the RoboEval benchmark, and perform a thorough analysis of the modes of failures, resulting in a taxonomy that highlights common pitfalls of LLMs at generating robot programs. We release our code and benchmark at https://amrl.cs.utexas.edu/codebotler/.

Figures (9)

• Figure 1: The system diagram of CodeBotler and RoboEval. CodeBotler receives a user prompt and queries a large language model (LLM) to generate a robot program (a). It can execute the program to send instructions to a robot via ROS Actionlib (c). Separately, RoboEval evaluates programs generated from its benchmark tasks using a symbolic simulator and a temporal trace evaluator to determine whether each program satisfies the task constraints or not (b).
• Figure 2: CodeBotler robot skills (a), RoboEval temporal logic (RTL) formula (b), and the LTL specifications vs. the RTL specifications of an example task (c). In section (c), the terms office, meet, and lunch are used to represent the regex patterns. The RTL specifications are simpler to express and have improved readability.
• Figure 3: The RoboEval benchmark includes 16 tasks, each with 5 prompt paraphrases. The figure displays these tasks' names and a detailed example of the task CountSavory.
• Figure 4: 5 LLMs are evaluated on the RoboEval benchmark. Each benchmark task contains 5 different prompt paraphrases and each bar represents the average pass@1 score of an LLM for generating responses across all 5 prompts within a given RoboEval benchmark task. Each error bar indicates the range from the highest to the lowest pass@1 score across all prompts. On the left side (a), the performance of LLMs on each task of the RoboEval benchmark is displayed. On the right side (b), the chart shows the performance of LLMs on tasks that have been adapted to exclusively evaluate the models' proficiency in performing commonsense reasoning.
• Figure 5: Cumulative Distribution Function (CDF) curves depict the percentage of prompts for which each LLM can generate correct LMPs at various pass@1 score thresholds. A perfect LLM would show a horizontal line at 100%, indicating it can generate correct LMPs for all prompts with a pass@1 score of 1. To maintain visual clarity, we limit the x-axis to $10^{-3}$ since all CDF plots eventually reach 100%.