Learning Symbolic and Subsymbolic Temporal Task Constraints from Bimanual Human Demonstrations

TL;DR

This work proposes a novel model-driven approach for the combined learning of symbolic and subsymbolic temporal task constraints from multiple bimanual human demonstrations and presents a subsymbolic foundation of a temporal task model that describes temporal nexuses of actions in the task based on distributions of temporal differences between semantic action keypoints.

Abstract

Learning task models of bimanual manipulation from human demonstration and their execution on a robot should take temporal constraints between actions into account. This includes constraints on (i) the symbolic level such as precedence relations or temporal overlap in the execution, and (ii) the subsymbolic level such as the duration of different actions, or their starting and end points in time. Such temporal constraints are crucial for temporal planning, reasoning, and the exact timing for the execution of bimanual actions on a bimanual robot. In our previous work, we addressed the learning of temporal task constraints on the symbolic level and demonstrated how a robot can leverage this knowledge to respond to failures during execution. In this work, we propose a novel model-driven approach for the combined learning of symbolic and subsymbolic temporal task constraints from multiple bimanual human demonstrations. Our main contributions are a subsymbolic foundation of a temporal task model that describes temporal nexuses of actions in the task based on distributions of temporal differences between semantic action keypoints, as well as a method based on fuzzy logic to derive symbolic temporal task constraints from this representation. This complements our previous work on learning comprehensive temporal task models by integrating symbolic and subsymbolic information based on a subsymbolic foundation, while still maintaining the symbolic expressiveness of our previous approach. We compare our proposed approach with our previous pure-symbolic approach and show that we can reproduce and even outperform it. Additionally, we show how the subsymbolic temporal task constraints can synchronize otherwise unimanual movement primitives for bimanual behavior on a humanoid robot.

Figures (4)

• Figure 1: Basic idea of our approach. Upper: Two modes of task execution where actions $x$ and $y$ either follow a before or meets pattern. Lower: Instead of formulating temporal task constraints between two actions $x$ and $y$ semantically using Allen relations (before or meets in this case), we represent them as Gaussian mixture models of temporal differences between action's starts and ends observed in human demonstrations. This allows for a combined subsymbolic and symbolic representation.
• Figure 2: To which degree a semantic temporal action keypoint $t_1$ is before, after or equal to another keypoint $t_2$ is decided by integrating the mixture density function $f_{M}$ of the GMM $M$ that models the distribution of temporal differences between $t_1$ and $t_2$. Red area: Degree of membership of $(t_1, t_2)$ in $\tilde{R}^\bullet_\text{before}$. Green area: Degree of membership of $(t_1, t_2)$ in $\tilde{R}^\bullet_\text{after}$. Blue area: Degree of membership of $(t_1, t_2)$ in $\tilde{R}^\bullet_\text{equals}$.
• Figure 3: Mean precision and recall with standard deviation of our new approach compared to our old approach dreher2022learning over $100$ simulated learning scenarios. Each learning scenario is an independent instantiation of a temporal task model that receives $100$ demonstrations of the synthetic dataset one by one in a unique order. The temporal task model of each learning scenario is evaluated after adding a new demonstration.
• Figure 4: Correct learning scenarios across $60$ demonstrations, comparing the new approach with that of our previous work dreher2022learning for both the synthetic dataset (Synthetic), as well as the KIT Bimanual Actions Dataset (Bimacs).