KiloBot: A Programming Language for Deploying Perception-Guided Industrial Manipulators at Scale

TL;DR

A Domain-Specific Language (DSL) for perception-guided manipulation applications that provides an easily accessible interface to construct and solve a sub-class of Task and Motion Planning (TAMP) problems that are important in practical applications and is mainly used by machine operators without coding experience in traditional programming languages.

Abstract

We would like industrial robots to handle unstructured environments with cameras and perception pipelines. In contrast to traditional industrial robots that replay offline-crafted trajectories, online behavior planning is required for these perception-guided industrial applications. Aside from perception and planning algorithms, deploying perception-guided manipulators also requires substantial effort in integration. One approach is writing scripts in a traditional language (such as Python) to construct the planning problem and perform integration with other algorithmic modules & external devices. While scripting in Python is feasible for a handful of robots and applications, deploying perception-guided manipulation at scale (e.g., more than 10000 robot workstations in over 2000 customer sites) becomes intractable. To resolve this challenge, we propose a Domain-Specific Language (DSL) for perception-guided manipulation applications. To scale up the deployment,our DSL provides: 1) an easily accessible interface to construct & solve a sub-class of Task and Motion Planning (TAMP) problems that are important in practical applications; and 2) a mechanism to implement flexible control flow to perform integration and address customized requirements of distinct industrial application. Combined with an intuitive graphical programming frontend, our DSL is mainly used by machine operators without coding experience in traditional programming languages. Within hours of training, operators are capable of orchestrating interesting sophisticated manipulation behaviors with our DSL. Extensive practical deployments demonstrate the efficacy of our method.

Figures (8)

• Figure 1: The graphical programming frontend for our programming language. Users orchestrate the robot behaviors by drag-and-drop programming to construct a control-flow graph, as shown in region (b) on the right. The region (a) on the left is the 3D visualization and resource window.
• Figure 2: Examples of control flow graph representation of user programs. (a) Nodes in a control flow graph correspond to a Python statements. (b) Edges in a control flow graph can be used to implement branch and loop structures, such as the green edge. (c) Similar to functions in Python, nodes in a control flow graph can be organized into routines, which can be invoked from another routine.
• Figure 3: A schematic example of robot pick-and-place behaviors and the corresponding user program in our DSL. Perception service provides a scene with two objects (A and B), each with two possible grasping poses (in dash lines), as shown in (a). The user program, as shown in (b), contains nodes that are intentionally undetermined offline and can not fully specify the robot behaviors. During execution, the interpreter invokes the behavior planning to make discrete decisions (e.g., selection of object instance and grasping pose) and generates robot trajectories. If the planning succeeds, the interpreter executes the user program given the planning result (online-parameters of each node). Please refer to Subsec. \ref{['subsec:example']} for a detailed explanation.
• Figure 4: Representative examples of discrete decisions in our DSL. (a) Select the object to pick from a set of perceived objects. (b) Select the grasping pose of the object-to-pick. (c) Decide the best IK solution for a given end-effector pose. (d) Choose the gripper tool from multiple candidates. (e) Select the target from a target set, which can be either automatically generated (e.g., in palletization applications) or explicitly specified by the user. (f) Select the symmetry of the object for a given object pose target.
• Figure 5: Branch structure types supported by a plan-routine. (a) The branch is explicitly marked as an exception. (b) The branch can be decided from data flow analysis. (c) The PlannerSelect branch used to set up decision-making problems regarding arbitrary factors, such as gripper tool selection in this example. Please refer to Subsec. \ref{['subsec:pr_struct']} for a detailed explanation.