RoboCritics: Enabling Reliable End-to-End LLM Robot Programming through Expert-Informed Critics

TL;DR

RoboCritics, an approach that augments LLM-based robot programming with expert-informed motion-level critics that enables more reliable and user-centered end-to-end robot programming with LLMs, is presented.

Abstract

End-user robot programming grants users the flexibility to re-task robots in situ, yet it remains challenging for novices due to the need for specialized robotics knowledge. Large Language Models (LLMs) hold the potential to lower the barrier to robot programming by enabling task specification through natural language. However, current LLM-based approaches generate opaque, "black-box" code that is difficult to verify or debug, creating tangible safety and reliability risks in physical systems. We present RoboCritics, an approach that augments LLM-based robot programming with expert-informed motion-level critics. These critics encode robotics expertise to analyze motion-level execution traces for issues such as joint speed violations, collisions, and unsafe end-effector poses. When violations are detected, critics surface transparent feedback and offer one-click fixes that forward structured messages back to the LLM, enabling iterative refinement while keeping users in the loop. We instantiated RoboCritics in a web-based interface connected to a UR3e robot and evaluated it in a between-subjects user study (n=18). Compared to a baseline LLM interface, RoboCritics reduced safety violations, improved execution quality, and shaped how participants verified and refined their programs. Our findings demonstrate that RoboCritics enables more reliable and user-centered end-to-end robot programming with LLMs.

Figures (4)

• Figure 1: RoboCritics workflow: (1) The user begins by providing a high-level task description. (2) The LLM generates a corresponding robot program. (3) The program is executed and evaluated against user-selected critics based on the resulting trajectory. (4) Feedback and results are stored in the interaction memory. (5) The LLM uses RAG to refine the program. (6) The refined program incorporates critic feedback. (7) The user validates the improved program via simulation. (8) Once verified, the final program is stored for future reference. (9) The validated program is deployed to the physical robot.
• Figure 2: User interface of RoboCritics: The panel on the left hosts a chat interface (a) for natural language interaction with the LLM agent. Users can chat by sending a message (i). The center panel exposes critics (b) allowing users to inspect and verify the generated program before execution. Users can activate critics by selecting them (g) and also fix the issues raised by critics by pressing the fix button (f). After selecting critics, users can execute the generated code by pressing "Run Code" button (d). The terminal shows the feedback from the selected critics (h). The panel on the right displays a simulation (c) of the generated robot program, allowing users to play and replay the execution as needed (e).
• Figure 3: Initial task state for each tasks performed in the user study. From left to right: recycling, sorting, and preparing breakfast.
• Figure 4: (left) Program quality scores across tasks for both with-critic (dark blue) and no-critic (light blue) conditions. Scores were computed using domain-specific critics, with higher scores indicating better program quality. Horizontal lines indicate statistically significant differences based on the Student's t-test ($p < .05^{\ast}$). Vertical lines in each bar graph indicate standard deviation. (right) Total number of critic activations across categories.