Transferring Kinesthetic Demonstrations across Diverse Objects for Manipulation Planning

TL;DR

The paper addresses transferring kinesthetic demonstrations of complex manipulation tasks to new task instances with different object geometries. It extends a screw-geometric planning framework by introducing geometry-aware motion-transfer frames that anchor task constraints to critical, geometry-informed locations and by transferring guiding poses using SE(3) transforms via ScLERP and the Jacobian pseudo-inverse. Key contributions include a procedure to identify critical locations, assign motion-transfer frames to demonstration and new-task objects, and preserve end-effector constraints during transfer while accommodating geometry changes, plus a computational criterion for task completion. Validation spans kinematic simulations across varied shapes and real hardware experiments, demonstrating improved collision avoidance and reduced spill risk when generalizing from a single demonstration to diverse objects in pouring tasks.

Abstract

Given a demonstration of a complex manipulation task such as pouring liquid from one container to another, we seek to generate a motion plan for a new task instance involving objects with different geometries. This is non-trivial since we need to simultaneously ensure that the implicit motion constraints are satisfied (glass held upright while moving), the motion is collision-free, and that the task is successful (e.g. liquid is poured into the target container). We solve this problem by identifying positions of critical locations and associating a reference frame (called motion transfer frames) on the manipulated object and the target, selected based on their geometries and the task at hand. By tracking and transferring the path of the motion transfer frames, we generate motion plans for arbitrary task instances with objects of different geometries and poses. We show results from simulation as well as robot experiments on physical objects to evaluate the effectiveness of our solution.

Figures (6)

• Figure 2: Pouring task involves pouring the content (secondary object) from the soup can (primary object) into the bowl (passive object).
• Figure 3: (a) Schematic sketch showing the overview of the motion segmentation algorithm described in mahalingam2023human for pouring the contents from a $\hbox{\it soup_can}$ into a $\hbox{\it bowl}$. (b) Simply transferring the constant screw segments (shown using dashed lines) extracted from the demonstration of pouring using a $\hbox{\it soup_can}$ to the execution of pouring using a $\hbox{\it box}$ results into a failure as the objects are colliding with each other.
• Figure 4: (b.1) Identifying the locations on the demonstrated objects critical for the task, and assigning the motion-transfer frames, $\left\{\sideset{_{}^{d}}{_{r}^{}}{\mathop C}\right\}$ and $\left\{\sideset{_{}^{d}}{_{s}^{}}{\mathop C}\right\}$ to the primary and passive objects used in the demonstration. (b.2) Assigning the motion-transfer frames, $\left\{\sideset{_{}^{n}}{_{r}^{}}{\mathop C}\right\}$ and $\left\{\sideset{_{}^{n}}{_{s}^{}}{\mathop C}\right\}$ to the primary and passive objects used in the new task instance.
• Figure 5: Successful pouring task executions in simulation with virtual objects (blue) of varying shapes and sizes using a single physical demonstration (red). Different values of $(a,b,n,h)$ are shown for the primary object.
• Figure 6: Two kinesthetic demonstrations of the pouring task using different primary and passive objects.