Latent Object Characteristics Recognition with Visual to Haptic-Audio Cross-modal Transfer Learning

TL;DR

This work proposes a cross-modal transfer learning approach from vision to haptic-audio, which facilitates the representation of object characteristics using indirect sensor data, thereby improving recognition accuracy.

Abstract

Recognising the characteristics of objects while a robot handles them is crucial for adjusting motions that ensure stable and efficient interactions with containers. Ahead of realising stable and efficient robot motions for handling/transferring the containers, this work aims to recognise the latent unobservable object characteristics. While vision is commonly used for object recognition by robots, it is ineffective for detecting hidden objects. However, recognising objects indirectly using other sensors is a challenging task. To address this challenge, we propose a cross-modal transfer learning approach from vision to haptic-audio. We initially train the model with vision, directly observing the target object. Subsequently, we transfer the latent space learned from vision to a second module, trained only with haptic-audio and motor data. This transfer learning framework facilitates the representation of object characteristics using indirect sensor data, thereby improving recognition accuracy. For evaluating the recognition accuracy of our proposed learning framework we selected shape, position, and orientation as the object characteristics. Finally, we demonstrate online recognition of both trained and untrained objects using the humanoid robot Nextage Open.

Figures (10)

• Figure 1: A rectangular prism typically offers greater stability compared to a cylinder or a sphere. The ease with which a cylinder rolls is contingent upon its orientation, while a sphere possesses the capability to roll in any direction.
• Figure 2: Overall learning model. We train the first phase vision module, and then transfer the latent space to the second phase haptic-audio and motor module to use it for initiating the training process of the latter module. Our objective is to discern object position, orientation, and shape utilising indirect information derived from haptic-audio cues.
• Figure 3: Nextage robot and sensor settings.
• Figure 4: 9 different shapes and sizes / weights objects for training.
• Figure 5: PCA on latent space after training the 1st phase vision module, which is transferred to the 2nd phase haptic-audio module. The latent space represents the object shape, orientation and position.