Towards Natural Language-Driven Assembly Using Foundation Models

TL;DR

This paper addresses the challenge of achieving reliable, language-driven control for industrial assembly tasks that demand high precision and nuanced contact dynamics. It introduces a global control policy based on Large Language Models (LLMs) that can steer behavior across tasks and dynamically switch between specialized skills trained for precise manipulation. The approach enables transfer to a finite set of defined skills, incorporating force/torque sensing and context-aware adjustments to handle contact, friction, and task-specific requirements. The work highlights the role of foundation models in both interpreting natural language and enhancing robotic control, potentially improving robustness and adaptability in automated assembly and disassembly workflows.

Abstract

Large Language Models (LLMs) and strong vision models have enabled rapid research and development in the field of Vision-Language-Action models that enable robotic control. The main objective of these methods is to develop a generalist policy that can control robots with various embodiments. However, in industrial robotic applications such as automated assembly and disassembly, some tasks, such as insertion, demand greater accuracy and involve intricate factors like contact engagement, friction handling, and refined motor skills. Implementing these skills using a generalist policy is challenging because these policies might integrate further sensory data, including force or torque measurements, for enhanced precision. In our method, we present a global control policy based on LLMs that can transfer the control policy to a finite set of skills that are specifically trained to perform high-precision tasks through dynamic context switching. The integration of LLMs into this framework underscores their significance in not only interpreting and processing language inputs but also in enriching the control mechanisms for diverse and intricate robotic operations.