Adaptive Whole-body Robotic Tool-use Learning on Low-rigidity Plastic-made Humanoids Using Vision and Tactile Sensors

TL;DR

This work tackles the challenge of modeling deflection in low-rigidity humanoids during tool-use, where tool grasping alters body posture and tool-tip position. It introduces the Whole-body Tool-use Network with Parametric Bias (WTNPB), a 7-layer encoder–decoder network that learns the mutual relations among joint angles, center of gravity, tool-tip position, and tool-tip image coordinates, with a 2D Parametric Bias to implicitly encode tool weight and length variations ($p \\in \\mathbb{R}^2$, $z \\in \\mathbb{R}^8$, input $\\bm{x}$ of dimension 17). Training is performed in two stages (simulation for initial learning of $W$ and $\\bm{p}_k$, then real-robot fine-tuning) and includes an online PB update mechanism that operates with fixed network weights to adapt to new tools. Control is achieved by gradient-based optimization of $\\bm{z}^{opt}$ to minimize a loss combining tool-tip and center-of-gravity accuracy, enabling robust whole-body tool-use on the KXR platform. The results show PB self-organization along tool-state axes, effective online adaptation with various sensor availabilities, and improved control accuracy after real-robot fine-tuning, demonstrating practical applicability to adaptive manipulation with low-rigidity robots.

Abstract

Various robots have been developed so far; however, we face challenges in modeling the low-rigidity bodies of some robots. In particular, the deflection of the body changes during tool-use due to object grasping, resulting in significant shifts in the tool-tip position and the body's center of gravity. Moreover, this deflection varies depending on the weight and length of the tool, making these models exceptionally complex. However, there is currently no control or learning method that takes all of these effects into account. In this study, we propose a method for constructing a neural network that describes the mutual relationship among joint angle, visual information, and tactile information from the feet. We aim to train this network using the actual robot data and utilize it for tool-tip control. Additionally, we employ Parametric Bias to capture changes in this mutual relationship caused by variations in the weight and length of tools, enabling us to understand the characteristics of the grasped tool from the current sensor information. We apply this approach to the whole-body tool-use on KXR, a low-rigidity plastic-made humanoid robot, to validate its effectiveness.

Figures (10)

• Figure 1: The concept of this study: learning the mutual relationship among joint angle, center of gravity, tool-tip position, and tool-tip screen coordinates for adaptive whole-body tool-use of low-rigidity robots considering the changes in tool weight and length.
• Figure 2: System overview of Whole-body Tool-use Network with Parametric Bias (WTNPB) including Data Collector, Network Trainer, Online Updater, and Tool-Tip Controller for Low-Rigidity Robots.
• Figure 3: Six types of tool states with various weights and lengths used in this study.
• Figure 4: The change in tool-tip position and center of gravity when handling tools with different weights.
• Figure 5: The trained parametric bias in the simulation experiment.