Creative Robot Tool Use with Large Language Models

TL;DR

Creative robot tool use is enabled by RoboTool, an LLM-driven architecture that converts natural-language task descriptions into executable code controlling parameterized robot skills. The approach uses four modules—Analyzer, Planner, Calculator, and Coder—to reason about implicit physical constraints, plan long-horizon tool use, compute skill parameters, and generate runnable code. A novel benchmark across two embodied platforms assesses tool selection, sequential tool use, and tool manufacturing, with RoboTool outperforming baselines in simulation and real-world experiments. The results show LLMs can capture 3D-physical constraints and generate creative tool use beyond conventional tool functions, expanding robot capabilities.

Abstract

Tool use is a hallmark of advanced intelligence, exemplified in both animal behavior and robotic capabilities. This paper investigates the feasibility of imbuing robots with the ability to creatively use tools in tasks that involve implicit physical constraints and long-term planning. Leveraging Large Language Models (LLMs), we develop RoboTool, a system that accepts natural language instructions and outputs executable code for controlling robots in both simulated and real-world environments. RoboTool incorporates four pivotal components: (i) an "Analyzer" that interprets natural language to discern key task-related concepts, (ii) a "Planner" that generates comprehensive strategies based on the language input and key concepts, (iii) a "Calculator" that computes parameters for each skill, and (iv) a "Coder" that translates these plans into executable Python code. Our results show that RoboTool can not only comprehend explicit or implicit physical constraints and environmental factors but also demonstrate creative tool use. Unlike traditional Task and Motion Planning (TAMP) methods that rely on explicit optimization, our LLM-based system offers a more flexible, efficient, and user-friendly solution for complex robotics tasks. Through extensive experiments, we validate that RoboTool is proficient in handling tasks that would otherwise be infeasible without the creative use of tools, thereby expanding the capabilities of robotic systems. Demos are available on our project page: https://creative-robotool.github.io/.

Figures (14)

• Figure 1: (a) Creative robot tool use with Large Language Models (RoboTool). RoboTool takes natural language descriptions as input, including the scene descriptions, environment- and embodiment-related constraints, and tasks. (b) We design a creative tool-use benchmark based on a quadrupedal robot and a robotic arm, including 6 challenging tasks that symbols three types of creative tool-use behaviors.
• Figure 2: Overview of our proposed RoboTool, which is a creative robot tool user consisting of four key components including Analyzer, Planner, Calculator and Coder.
• Figure 3: Analyzer output.
• Figure 4: Planner output.
• Figure 5: Visualization of RoboTool's creative tool-use behaviors. (a) Tool selection. The quadrupedal robot needs to select the surfboard over a strip of cloth and push it to bridge the gap (Sofa-Traversing). The robotic arm needs to choose the hammer among many options and use it as a hook to pull the milk carton into the workspace (Milk-Reaching). (b) Sequential tool use. The quadrupedal robot needs to push a small box against a large box adjacent to the sofa and use the small box as the initial stepstone and the large box as the intermediate stepstone to climb onto the sofa (Sofa-Climbing). The robotic arm needs to pick up a stick, push a can onto a strip of paper, and then pull the paper closer with the can on it (Can-Grasping). (c) Tool manufacturing. The quadrupedal needs to identify the hidden lever structure in the environment and push away a chair supporting one end of the lever so that it can activate the lever arm and lift a heavy cube (Cube-Lifting). The robotic arm needs to assemble magnetic blocks to create a stick to press a button outside its workspace (Button-Pressing).