Information-Theoretic Detection of Bimanual Interactions for Dual-Arm Robot Plan Generation

TL;DR

This paper tackles the problem of programming dual-arm robots from a single RGB video by introducing an information-theoretic framework that analyzes per-frame scene graphs to infer hand coordination. It combines mutual information and co-information metrics with graph topologies to identify coordination modes, segments the task into Interaction Units, and maps results to a modular Behavior Tree that guides dual-arm execution. The method is validated on open HANDSOME data and the KIT Bimanual dataset, showing robust coordination detection, BT generation, and accurate robot replication, with improvements over a state-of-the-art baseline in plan generation. The approach enables one-shot PbD for bimanual tasks and holds practical impact for flexible, end-user-friendly dual-arm robotics, with future work targeting multi-demo optimization and trajectory learning integration.

Abstract

Programming by demonstration is a strategy to simplify the robot programming process for non-experts via human demonstrations. However, its adoption for bimanual tasks is an underexplored problem due to the complexity of hand coordination, which also hinders data recording. This paper presents a novel one-shot method for processing a single RGB video of a bimanual task demonstration to generate an execution plan for a dual-arm robotic system. To detect hand coordination policies, we apply Shannon's information theory to analyze the information flow between scene elements and leverage scene graph properties. The generated plan is a modular behavior tree that assumes different structures based on the desired arms coordination. We validated the effectiveness of this framework through multiple subject video demonstrations, which we collected and made open-source, and exploiting data from an external, publicly available dataset. Comparisons with existing methods revealed significant improvements in generating a centralized execution plan for coordinating two-arm systems.

Figures (11)

• Figure 1: Framework overview. The first block maps each video frame $k$ into graphs $G_R[k], G_L[k]$ capturing task-relevant interactions for each hand, while the second block translates these into robot instructions, identifying arm coordination mode, extracting action sequences, and generating a dual-arm execution plan.
• Figure 2: Possible topologies for $G_R$ and $G_L$ encoding single-hand interactions: (A) hand $h$ interacting with the manipulated object $o_m$; (B) hand manipulating a unity of three objects $u_m$; (C) interaction between $o_m$ and a static background object $o_{bkg}$; (D) $u_m$ interacting with $o_{bkg}$.
• Figure 3: Segmentation of a synchronous bimanual activity. Each retrieved IU is represented by its $G^\text{repr}$. Eq. \ref{['eq:gdiff']} is applied to adjacent $G^\text{repr}$ obtaining $^\text{diff}G^\text{repr}$, which includes sub-graphs encoding new (positive sign) and ended (negative sign) interactions. Each sub-graph is mapped into high-level primitives $\psi$.
• Figure 4: Subtree for uncoordinated dual-arm activities, detailing the structure of one arm subtree in the blue box. The subtree handling movements towards a target object is detailed as well, in the red box.
• Figure 5: Subtree for a synchronous dual-arm activity.