Learning Alternative Ways of Performing a Task

TL;DR

The work tackles learning multiple execution strategies from a small set of positive demonstrations by introducing a graph-based inductive framework. It represents each task as a dependency graph $G=(V,E)$ with start $v_S$ and finish $v_F$, and uses an aggregation/refinement loop (MMDG) to extract several distinct, interpretable representations that cover observed executions while reducing noise. Experimental validation on MIS suturing data and brownie cooking shows the method can isolate different execution styles and provide auditable supervision cues, with fitness on unseen traces generally in the 60–70% range and substantial simplification of the learned representations. This approach offers a practical, data-efficient alternative to single-pattern process models in data-scarce domains and lays groundwork for extensions with natural-language task descriptions and interactive supervision tools.

Abstract

A common way of learning to perform a task is to observe how it is carried out by experts. However, it is well known that for most tasks there is no unique way to perform them. This is especially noticeable the more complex the task is because factors such as the skill or the know-how of the expert may well affect the way she solves the task. In addition, learning from experts also suffers of having a small set of training examples generally coming from several experts (since experts are usually a limited and expensive resource), being all of them positive examples (i.e. examples that represent successful executions of the task). Traditional machine learning techniques are not useful in such scenarios, as they require extensive training data. Starting from very few executions of the task presented as activity sequences, we introduce a novel inductive approach for learning multiple models, with each one representing an alternative strategy of performing a task. By an iterative process based on generalisation and specialisation, we learn the underlying patterns that capture the different styles of performing a task exhibited by the examples. We illustrate our approach on two common activity recognition tasks: a surgical skills training task and a cooking domain. We evaluate the inferred models with respect to two metrics that measure how well the models represent the examples and capture the different forms of executing a task showed by the examples. We compare our results with the traditional process mining approach and show that a small set of meaningful examples is enough to obtain patterns that capture the different strategies that are followed to solve the tasks.

Figures (13)

• Figure 1: The pipeline to learn the task models.
• Figure 2: A sequence of activities (on the top) expressed as a dependency graph (on the bottom). The vertices corresponding to the synthetic activities (S and F) are highlighted in bold and shadowed, and the activities that appear more than once are depicted in colour.
• Figure 3: Effect of applying different thresholds for refining a $G^+$ formed by the aggregation of 10 dependency graphs (shown on the top). Note that the refinement procedure keeps all the edges of $G^+$ whose weight is greater than $\theta$.
• Figure 4: Overlap checking between a threshold graph $G^{+}_\theta$ and two dependency graphs $G_1$ and $G_2$. There is no overlapping between $G^{+}_\theta$ and $G_1$ because their intersection $G_1 \cap G^{+}_\theta$ (on the left bottom) does not hold the validity constraint ($V_S \neq V_F$). In contrast, $G^{+}_\theta$ overlaps with $G_2$ because their intersected graph $G_2 \cap G^{+}_\theta$ (on the right bottom) is valid ($V_S = V_F$).
• Figure 5: Example of a suturing cycle represented by frame steps. From left to right, frame (1) shows the needle insertion in the input dot on the tissue, frame (2) represents the needle grasping on the output dot from the other side of the tissue, and frame (3) shows the needle transferring from left tool to the right tool. Captures obtained from JIGSAWS dataset gao2014jhu.

Theorems & Definitions (6)

• Definition 1
• Definition 2
• Definition 3
• Definition 4
• Definition 5
• Definition 6