Task-sequencing Simulator: Integrated Machine Learning to Execution Simulation for Robot Manipulation

TL;DR

The paper addresses the challenge of unifying simulation-for-learning and simulation-for-execution in multi-step robotic manipulation. It proposes a task-sequencing simulator built on concept models that decompose complex tasks into reusable blocks, enabling seamless transitions from training to real-world execution. A two-layer architecture—Concept Interface for task sequencing and Environment Engine Pipeline for state observation—supports integrating various engines and learning paradigms. Demonstrations on grasping, door-opening, and bringing show how learned blocks can be reused across scenarios and transferred from simulation to real robots with minimal sim-to-real gaps.

Abstract

A task-sequencing simulator in robotics manipulation to integrate simulation-for-learning and simulation-for-execution is introduced. Unlike existing machine-learning simulation where a non-decomposed simulation is used to simulate a training scenario, the task-sequencing simulator runs a composed simulation using building blocks. This way, the simulation-for-learning is structured similarly to a multi-step simulation-for-execution. To compose both learning and execution scenarios, a unified trainable-and-composable description of blocks called a concept model is proposed and used. Using the simulator design and concept models, a reusable simulator for learning different tasks, a common-ground system for learning-to-execution, simulation-to-real is achieved and shown.

Figures (6)

• Figure 1: The proposed task-sequencing simulator which enables scenario composition for both learning and execution in robotics manipulation.
• Figure 2: An illustration of a task model.
• Figure 3: An illustration of a concept model.
• Figure 4: Example concept models of eight different tasks in the screw-theory based classification.
• Figure 5: Results of the task-sequencing simulator when used for learning.