Correspondence learning between morphologically different robots via task demonstrations

TL;DR

The paper tackles cross-m morphology skill transfer by learning sensorimotor correspondences between heterogeneous robots. It introduces a framework that uses coupled CNMPs to build a common latent representation $L$ via convex blending $L = \sum_r p^r L^r$, enabling a single robot's task demonstrations to generate trajectories for other robots without additional hyperparameters. Empirical results across same-path, different-path, and varying-SM-space tasks demonstrate accurate cross-robot trajectory generation and scalability to multiple morphologies, including a real-world proof-of-concept. This approach advances zero-shot transfer in robotic skill learning, with practical impact for scalable multi-robot systems, though it leaves room for incorporating object features and more diverse robot types in future work.

Abstract

We observe a large variety of robots in terms of their bodies, sensors, and actuators. Given the commonalities in the skill sets, teaching each skill to each different robot independently is inefficient and not scalable when the large variety in the robotic landscape is considered. If we can learn the correspondences between the sensorimotor spaces of different robots, we can expect a skill that is learned in one robot can be more directly and easily transferred to other robots. In this paper, we propose a method to learn correspondences among two or more robots that may have different morphologies. To be specific, besides robots with similar morphologies with different degrees of freedom, we show that a fixed-based manipulator robot with joint control and a differential drive mobile robot can be addressed within the proposed framework. To set up the correspondence among the robots considered, an initial base task is demonstrated to the robots to achieve the same goal. Then, a common latent representation is learned along with the individual robot policies for achieving the goal. After the initial learning stage, the observation of a new task execution by one robot becomes sufficient to generate a latent space representation pertaining to the other robots to achieve the same task. We verified our system in a set of experiments where the correspondence between robots is learned (1) when the robots need to follow the same paths to achieve the same task, (2) when the robots need to follow different trajectories to achieve the same task, and (3) when complexities of the required sensorimotor trajectories are different for the robots. We also provide a proof-of-the-concept realization of correspondence learning between a real manipulator robot and a simulated mobile robot.

Figures (12)

• Figure 1: The conceptual summary of the task-based correspondence learning problems studied in this paper. The middle, left, and right columns represent task and sensorimotor (SM) spaces of the mobile and manipulator robots, respectively. The shapes of the markers represent tasks. Each marker in the middle plot corresponds to the required robot execution trajectory in the task space to achieve the task. Each marker in the left and right plots represents the required SM trajectory of the corresponding robot to achieve the task. The dashed line between markers represents the mapping between SM and task spaces. In (A), the robots follow the same task-space trajectory to achieve the same task. In (B), the robots need to follow different trajectories to achieve the same task. In (C), the robots again need to follow different trajectories, but in this case, the complexity of the sensorimotor mappings is also different.
• Figure 2: The general overview: Task-based correspondence via forming common latent representation between different robots.$E$ represents encoders, $S$ represents sampled observations, $L$ values represent latent representations and $Q$ represents decoders. Given observations from an SM trajectory of one robot, our system can generate the full SM trajectory for the other robot to achieve the same task by setting the blending weight p to 0 or 1. Please refer to the text for the details of training and generation.
• Figure 3: Training and test paths correspond to the paths required to achieve the obstacle avoidance tasks in Section \ref{['A']}. Left and right figures show the manipulator and mobile robots. Blue and red paths were used for training and testing, respectively. Green lines show the action range of the manipulator.
• Figure 4: The snapshots from the mobile robot to the manipulator (on top) and from the manipulator to the mobile robot (on bottom) for the experiment in Section \ref{['A']}. Using three observations from the given trajectory on a new task configuration (on the left of the figure), the system is able to generate the desired trajectories for the other robot (on the right ).
• Figure 5: Generated trajectories of one of the data sets of experiment in Section \ref{['A']} with their ground truth values. Dashed lines are expected ground truth values and solid lines represent the generated values. The upper part shows the results obtained using our model while the lower part shows the results obtained using the ACNMP approach.