Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unified Learning of Temporal Task Structure and Action Timing for Bimanual Robot Manipulation

Christian Dreher, Patrick Dormanns, Andre Meixner, Tamim Asfour

TL;DR

This work presents an approach for learning both symbolic and subsymbolic temporal task constraints from human demonstrations and deriving executable, temporally parametrized plans for bimanual manipulation.

Abstract

Temporal task structure is fundamental for bimanual manipulation: a robot must not only know that one action precedes or overlaps another, but also when each action should occur and how long it should take. While symbolic temporal relations enable high-level reasoning about task structure and alternative execution sequences, concrete timing parameters are equally essential for coordinating two hands at the execution level. Existing approaches address these two levels in isolation, leaving a gap between high-level task planning and low-level movement synchronization. This work presents an approach for learning both symbolic and subsymbolic temporal task constraints from human demonstrations and deriving executable, temporally parametrized plans for bimanual manipulation. Our contributions are (i) a 3-dimensional representation of timings between two actions with methods based on multivariate Gaussian Mixture Models to represent temporal relationships between actions on a subsymbolic level, (ii) a method based on the Davis-Putnam-Logemann-Loveland (DPLL) algorithm that finds and ranks all contradiction-free assignments of Allen relations to action pairs, representing different modes of a task, and (iii) an optimization-based planning system that combines the identified symbolic and subsymbolic temporal task constraints to derive temporally parametrized plans for robot execution. We evaluate our approach on several datasets, demonstrating that our method generates temporally parametrized plans closer to human demonstrations than the most characteristic demonstration baseline.

Unified Learning of Temporal Task Structure and Action Timing for Bimanual Robot Manipulation

TL;DR

This work presents an approach for learning both symbolic and subsymbolic temporal task constraints from human demonstrations and deriving executable, temporally parametrized plans for bimanual manipulation.

Abstract

Temporal task structure is fundamental for bimanual manipulation: a robot must not only know that one action precedes or overlaps another, but also when each action should occur and how long it should take. While symbolic temporal relations enable high-level reasoning about task structure and alternative execution sequences, concrete timing parameters are equally essential for coordinating two hands at the execution level. Existing approaches address these two levels in isolation, leaving a gap between high-level task planning and low-level movement synchronization. This work presents an approach for learning both symbolic and subsymbolic temporal task constraints from human demonstrations and deriving executable, temporally parametrized plans for bimanual manipulation. Our contributions are (i) a 3-dimensional representation of timings between two actions with methods based on multivariate Gaussian Mixture Models to represent temporal relationships between actions on a subsymbolic level, (ii) a method based on the Davis-Putnam-Logemann-Loveland (DPLL) algorithm that finds and ranks all contradiction-free assignments of Allen relations to action pairs, representing different modes of a task, and (iii) an optimization-based planning system that combines the identified symbolic and subsymbolic temporal task constraints to derive temporally parametrized plans for robot execution. We evaluate our approach on several datasets, demonstrating that our method generates temporally parametrized plans closer to human demonstrations than the most characteristic demonstration baseline.
Paper Structure (29 sections, 6 equations, 7 figures, 4 algorithms)

This paper contains 29 sections, 6 equations, 7 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. High-level: Task Modes and Temporal Task Structure
  4. Precedence Relations
  5. Complex Relations
  6. Low-level: MP Synchronization and Coupling
  7. Emerging Synchronization (Bottom-Up)
  8. Assigned Synchronization (Top-Down)
  9. Problem Statement
  10. Definition of Task, Action, Demonstration
  11. Definition of Task Assignments
  12. Definition of Timing
  13. Definition of Plan and Temporally Parametrized Plan
  14. Approach
  15. Temporal Relationship Assessment
  16. ...and 14 more sections

Figures (7)

  • Figure 1: Overview of our approach and contributions: Temporal Relationship Assessment, Temporal Task Constraint Inference, and Temporal Planning.
  • Figure 2: Example of our proposed timing space, depicting the relationship between two actions. Black points are observed timings between the two actions from several human demonstrations. Ellipsoids denote modes of the multivariate .
  • Figure 3: Visualization of selected Allen relations in $\mathbb{T}^3$. Points in the corresponding regions can be attributed to the respective Allen relation.
  • Figure 4: Example of conditioning a timing model to a given Allen relation. Grey ellipses depict the modes of the . The blue area shows the Allen relation finished by in timing space the timing model should be conditioned to. The orange point shows the maximum of the 's probability density function after restricting it to the finished by area.
  • Figure 5: Fully parametrized plans of two complex tasks from the bimacs generated by our approach. Task 5, "prepare muesli" on the left side, and Task 6, "disassemble component" on the right side. Colors depict identified subtasks, with purple being an exception to show actions can that span subtasks.
  • ...and 2 more figures