Programming Manipulators by Instructions

TL;DR

This paper addresses the challenge of programming robots by non-experts and in field settings by introducing an instructions-based paradigm that uses a scripting interface to orchestrate a library of base skills via a behavior-tree task controller. A skill-centric knowledge graph stores robot state, environment context, and instruction history, enabling transformation of instruction sequences into reusable composite skills. The method introduces a Custom Skill node to sequence arbitrary base skills and demonstrates the system in a Drake-based pick-and-place simulation with remote programming capabilities and a Python API. The results illustrate practical end-to-end instruction-based programming, hinting at scalable skill acquisition and remote operation in real-world domains.

Abstract

We propose an instructions-based approach for robot programming where the programmer interacts with the robot by issuing simple commands in a scripting language, like python. Internally, these commands make use of pre-programmed motion and manipulation skills coordinated by a behaviour tree task controller. A knowledge graph keeps track of the state of the robot and the environment and of all the instructions given to the robot by the programmer. This allows to easily transform sequences of instructions into new skills that can be reused in the same or in other tasks. An advantage of this approach is that the programmer does not need to be located physically next to the robot, but can work remotely, either with a physical robot or with a digital twin. To demonstrate the concept, we show an interactive simulation of a robot manipulator in a pick and place scenario.

Figures (5)

• Figure 1: Overview of system. The user interacts with a robot by adding new task event nodes to a knowledge graph. These are contextualized by automatically linking them to existing agents and objects in the KG. A task controller periodically polls the KG for new task events, extracts the necessary context information to setup the skills required and coordinates the execution by the robot using ROS.
• Figure 2: Base nodes and relations in Skills-centered Knowledge Graph.
• Figure 3: Snippet of a Pick and Place skills library stored in a Knowledge Graph.
• Figure 4: Behaviour tree with base skills and custom skill. The Chooser sub-tree returns Success when the xor condition is satisfied. The Custom skill sub-tree is ticked only in case the Chooser sub-tree returns Failure.
• Figure 5: BT of base skill.