Learning Semantic-Geometric Task Graph-Representations from Human Demonstrations

TL;DR

The paper tackles learning long-horizon bimanual manipulation by jointly modeling discrete action semantics and continuous geometric evolution. It introduces semantic-geometric task graphs and an encoder–decoder architecture (MPNN encoder + Transformer decoder) that predicts future actions, objects, and motions over extended horizons, with transfer to a physical robot. Key contributions include the graph representation capturing object identities and relations, the joint prediction framework with action chunking and temporal ensembles, and extensive validation on human demonstrations and robot transfer. The results demonstrate improved generalization in highly variable tasks and highlight the potential of reusable task abstractions for downstream manipulation planning.

Abstract

Learning structured task representations from human demonstrations is essential for understanding long-horizon manipulation behaviors, particularly in bimanual settings where action ordering, object involvement, and interaction geometry can vary significantly. A key challenge lies in jointly capturing the discrete semantic structure of tasks and the temporal evolution of object-centric geometric relations in a form that supports reasoning over task progression. In this work, we introduce a semantic-geometric task graph-representation that encodes object identities, inter-object relations, and their temporal geometric evolution from human demonstrations. Building on this formulation, we propose a learning framework that combines a Message Passing Neural Network (MPNN) encoder with a Transformer-based decoder, decoupling scene representation learning from action-conditioned reasoning about task progression. The encoder operates solely on temporal scene graphs to learn structured representations, while the decoder conditions on action-context to predict future action sequences, associated objects, and object motions over extended time horizons. Through extensive evaluation on human demonstration datasets, we show that semantic-geometric task graph-representations are particularly beneficial for tasks with high action and object variability, where simpler sequence-based models struggle to capture task progression. Finally, we demonstrate that task graph representations can be transferred to a physical bimanual robot and used for online action selection, highlighting their potential as reusable task abstractions for downstream decision-making in manipulation systems.

Figures (4)

• Figure 1: Graph model architecture: the graph encoder transforms features into embeddings, MPNN learns graph embeddings, and prediction heads forecast actions, objects, and motions.
• Figure 2: Predictions on a cooking task of the KIT Bimacs Dataset using the observations of just 100 timesteps (denoted by the vertical dashed line). The first three plots show the 3D motions for the different objects in the scene. The last plot shows the action and action-object predictions for both hands.
• Figure 3: Accuracies for the different action and object predictions of the different models on the various datasets (higher values are better).
• Figure 4: Results of the RMSE for motion prediction of different models on the various datasets (lower is better).