MAkEable: Memory-centered and Affordance-based Task Execution Framework for Transferable Mobile Manipulation Skills

TL;DR

MkEable is presented, a versatile uni- and multi-manual mobile manipulation framework that facilitates the transfer of capabilities and knowledge across different tasks, environments, and robots and its applicability is demonstrated in real-world experiments for multiple robots, tasks, and environments.

Abstract

To perform versatile mobile manipulation tasks in human-centered environments, the ability to efficiently transfer learned tasks and experiences from one robot to another or across different environments is key. In this paper, we present MAkEable, a versatile uni- and multi-manual mobile manipulation framework that facilitates the transfer of capabilities and knowledge across different tasks, environments, and robots. Our framework integrates an affordance-based task description into the memory-centric cognitive architecture of the ARMAR humanoid robot family, which supports the sharing of experiences and demonstrations for transfer learning. By representing mobile manipulation actions through affordances, i.e., interaction possibilities of the robot with its environment, we provide a unifying framework for the autonomous uni- and multi-manual manipulation of known and unknown objects in various environments. We demonstrate the applicability of the framework in real-world experiments for multiple robots, tasks, and environments. This includes grasping known and unknown objects, object placing, bimanual object grasping, memory-enabled skill transfer in a drawer opening scenario across two different humanoid robots, and a pouring task learned from human demonstration.

Figures (11)

• Figure 1: Our framework facilitates triple-mode transfer, i. e., across tasks (e. g., grasp and place), robots (e. g., ARMAR-6 and ARMAR-DE) and environments (e. g., household and maintenance).
• Figure 2: Embedding of MAkEable into the memory-centric cognitive architecture Peller2023 implemented in ArmarX. Several strategies that implement the five steps of the architecture (see \ref{['sec:system_architecture']}) are connected to the robot's memory.
• Figure 3: Simplified class diagram of the IDF task description. Types marked with a "*" are optional.
• Figure 4: Data flow in our framework visualizing the interaction of the IDF task description with the system architecture.
• Figure 5: Action hypothesis extraction for unknown objects. Point cloud segmentation using Segment Anything kirillov2023segment (left) and object-oriented bounding boxes fitting Grimm2021 (right).