Cocobo: Exploring Large Language Models as the Engine for End-User Robot Programming

TL;DR

Cocobo addresses the challenge of enabling non-programmers to customize service robots by integrating large language models with a dual NL-conversational and flowchart-based interface. It introduces an LLM-driven pipeline that translates natural language intents into executable robot programs, with mechanisms for generating explanations and converting between code and flowcharts, including a MagicDebug debugging mode. In a user study with 16 participants, Cocobo demonstrated good usability and a natural conversational experience, though it revealed issues with output stability and content length that affect responsiveness. The work shows promise for practical end-user robot programming and highlights avenues for broader API support, real-world testing, and improved prompt design to better handle non-expert users.

Abstract

End-user development allows everyday users to tailor service robots or applications to their needs. One user-friendly approach is natural language programming. However, it encounters challenges such as an expansive user expression space and limited support for debugging and editing, which restrict its application in end-user programming. The emergence of large language models (LLMs) offers promising avenues for the translation and interpretation between human language instructions and the code executed by robots, but their application in end-user programming systems requires further study. We introduce Cocobo, a natural language programming system with interactive diagrams powered by LLMs. Cocobo employs LLMs to understand users' authoring intentions, generate and explain robot programs, and facilitate the conversion between executable code and flowchart representations. Our user study shows that Cocobo has a low learning curve, enabling even users with zero coding experience to customize robot programs successfully.

Figures (6)

• Figure 1: Overview of Cocobo, an LLM-powered end-user programming system for service robot task customization. The system’s UI consists of two views: (A) Flowchart View, which is composed of several components including (A1) node library, (A2) interactive flowchart, (A3) node properties panel, and (A4) buttons group; (B) The Conversational View enables natural language conversational interaction between users and the system.
• Figure 2: Users interact with Cocobo through multiple rounds of dialogues to communicate and customize their requirements. This interaction further facilitates the generation of code, as well as the creation of accompanying textual explanations and flowchart representations.
• Figure 3: Users can directly modify their requirements through text input to alter the code. Cocobo will interpret these modifications and regenerate the flowchart accordingly.
• Figure 4: An illustration of the flowchart's editing capabilities allows users to add and remove nodes, as well as modify the connections between them. Within the node properties panel, users can modify the robot behaviors represented by each node using natural language.
• Figure 5: An illustration of the usage process for the Magic Debug feature: After selecting one or multiple nodes, users click the Magic Debug button(Figure \ref{['fig: overview']}, A4). The system then enters node debugging mode, allowing users to inspect and modify the selected nodes through the conversational UI.