A Combined Learning and Optimization Framework to Transfer Human Whole-body Loco-manipulation Skills to Mobile Manipulators

TL;DR

This work tackles transferring human loco-manipulation skills to mobile manipulators by coupling a learning module with a hierarchical optimization controller. It maps human wrist/pelvis motions to the end-effector and base, learns a whole-body trajectory with Kernelized Movement Primitive (KMP) and generalizes to new targets, then uses Hierarchical Quadratic Programming (HQP) to generate feasible joint commands via a Stack of Tasks, prioritizing end-effector tracking over base pose. Experiments on the MOCA platform demonstrate successful replication and generalization of a locomotion-integrated pick-and-place task with non-zero contact velocity, highlighting the framework’s robustness to geometric differences. The approach offers a scalable path for fluent loco-manipulation in mobile manipulators and sets the stage for broader applications and reactive extensions.

Abstract

Humans' ability to smoothly switch between locomotion and manipulation is a remarkable feature of sensorimotor coordination. Leaning and replication of such human-like strategies can lead to the development of more sophisticated robots capable of performing complex whole-body tasks in real-world environments. To this end, this paper proposes a combined learning and optimization framework for transferring human's loco-manipulation soft-switching skills to mobile manipulators. The methodology departs from data collection of human demonstrations for a locomotion-integrated manipulation task through a vision system. Next, the wrist and pelvis motions are mapped to mobile manipulators' End-Effector (EE) and mobile base. A kernelized movement primitive algorithm learns the wrist and pelvis trajectories and generalizes to new desired points according to task requirements. Next, the reference trajectories are sent to a hierarchical quadratic programming controller, where the EE and the mobile base reference trajectories are provided as the first and second priority tasks, generating the feasible and optimal joint level commands. A locomotion-integrated pick-and-place task is executed to validate the proposed approach. After a human demonstrates the task, a mobile manipulator executes the task with the same and new settings, grasping a bottle at non-zero velocity. The results showed that the proposed approach successfully transfers the human loco-manipulation skills to mobile manipulators, even with different geometry.

Figures (6)

• Figure 1: Human whole-body motion behavior for a locomotion-integrated pick-and-place task: the blue target object is outside the workspace of the right arm. Thus, the human first approaches the object and subsequently manipulates it in a single smooth motion.
• Figure 2: Overall proposed framework. From left to right, inside the whole-body trajectory learning module, the reference trajectories are extracted from human demonstration by GMMs/GMR. After the new desired points are given, KMP generalizes the learned skills to new settings. Next, the desired EE pose, velocity, and mobile base pose are sent to the HQP controller, which computes the optimal joint trajectories based on the hierarchical SoT and imposed constraints. These are passed to the robotic arm's lower-level joint impedance controller and to the mobile base velocity controller. The corresponding equations are also presented in this figure.
• Figure 3: Experimental setting for human demonstration. The subject was asked to naturally pick the bottle at point $A$ while walking at a natural speed and to place it at point $B$. All the motions were described in the world frame $\bm{\Sigma_{W}}$. The frames of the subject and MOCA are illustrated.
• Figure 4: Snapshots of the experiments. (top) Human whole-body demonstration collection for a bottle pick-and-place task. (middle) Experiment 1: autonomous task execution in nominal conditions. (bottom) Experiment 2: autonomous task execution with new randomized relative distance, where the initial EE and base positions were changed.
• Figure 5: Results of the replica task in nominal conditions with MOCA. From top to bottom, first row: EE desired position $\bm{x}_d \in\mathbb{R}^{3}$ and actual position $\bm{x}_a \in\mathbb{R}^{3}$ in the world frame $\bm{\Sigma_W}$; Second row: EE actual position in the robotic arm base frame $\bm{\Sigma_R}$; Third row: desired mobile base learned pose $\bm{q}_{b,d}$, optimal pose $\bm{q}_b^*$ from HQP, and actual pose $\bm{q}_{b,a}$; Fourth row: EE desired velocity $\dot{\bm{x}}_d \in\mathbb{R}^{3}$ and actual velocity $\dot{\bm{x}}_a \in\mathbb{R}^{3}$. At $t=0$, the magenta circles marked the initial states $\bm{x}_d=\ [ 0.417 \quad 0.062\quad 1.107\ ]^T~m$, $\bm{q}_{b,d}=\ [ -0.288~m\quad 0.263~m\quad 0.046~rad\ ]^T$, $\dot{\bm{x}}_d=\ [0\quad 0\quad 0\ ]^T~m/s$; At $t=18.15~s$, the desired grasp point (bottle position) states are highlighted in red circles: $\bm{x}_d=\ [ 3.687 \quad 0.423\quad 0.812\ ]^T~m$, $\bm{q}_{b,d}=\ [ 2.823~m\quad 0.427~m\quad 0.885~rad\ ]^T$, $\dot{\bm{x}}_d=\ [0.056\quad 0.164\quad 0.072\ ]^T ~m/s$.