Toward Automated Programming for Robotic Assembly Using ChatGPT

TL;DR

This work addresses the labor-intensive and specialized nature of programming robots for adaptive assembly by leveraging ChatGPT to automate task decomposition, code generation, and testing in simulation. It introduces a two-agent architecture (Task Decomposition Agent and Script Generation Agent) that operates on CAD-derived data fed as JSON, producing and refining Python scripts for robotic tasks within a simulated workcell. The approach demonstrates end-to-end capabilities on tasks such as gripper selection, script debugging, and a skateboard-truck assembly, while acknowledging limitations in perception-rich reasoning and the need for human oversight. The findings highlight the potential of LLM-assisted robotics programming to reduce development time and enable broader adaptability, with future directions including finer spatial reasoning and data-driven fine-tuning to improve robustness and generalization.

Abstract

Despite significant technological advancements, the process of programming robots for adaptive assembly remains labor-intensive, demanding expertise in multiple domains and often resulting in task-specific, inflexible code. This work explores the potential of Large Language Models (LLMs), like ChatGPT, to automate this process, leveraging their ability to understand natural language instructions, generalize examples to new tasks, and write code. In this paper, we suggest how these abilities can be harnessed and applied to real-world challenges in the manufacturing industry. We present a novel system that uses ChatGPT to automate the process of programming robots for adaptive assembly by decomposing complex tasks into simpler subtasks, generating robot control code, executing the code in a simulated workcell, and debugging syntax and control errors, such as collisions. We outline the architecture of this system and strategies for task decomposition and code generation. Finally, we demonstrate how our system can autonomously program robots for various assembly tasks in a real-world project.

Figures (10)

• Figure 1: Our workflow utilizes GPT-4's generalization and code-writing abilities to contextualize robotic workcell and CAD information in order to generate code for assembly tasks such as "Assemble the Skateboard Truck".
• Figure 2: Example of demonstration provided for few-shot prompting. Provides context on how language output from ChatGPT should be formatted and what a "successful" example looks like. Formatting inspiration taken from VoxPoser voxposer and Code as Policies c2
• Figure 3: Example TDA input/output. User inputs a task query and the output is a structured list of subtasks each with an assigned robotic behavior primitive. In addition to the task query, information about the assembly, objects that reside in it, grippers, robots etc. are also provided as context during input.
• Figure 4: Script Generation Agent class architecture. The agent history gets passed to a client that communicates with ChatGPT to generate a script. The script is added to the history and executed in simulation. Then feedback is added to the history and the process repeats.
• Figure 5: Geometry and assembly representations of the skateboard truck.