Design and Benchmarking of A Multi-Modality Sensor for Robotic Manipulation with GAN-Based Cross-Modality Interpretation

TL;DR

This work introduces ViTacTip, a compact multi-modality sensor that fuses vision, tactile, proximity, and force sensing via a see-through-skin and biomimetic pins. A GAN-based modality-switching framework (MR-GAN and LR-GAN) enables cross-modality interpretation, allowing ViTacTip to transition between ViTac, TacTip, and ViTacTip-style data without hardware switches. The paper provides extensive hardware benchmarking, including contact-point detection, pose regression, grating identification, and a hierarchical multi-task network for hardness, material, and texture recognition, demonstrating superior performance over single-modality baselines. The results suggest ViTacTip offers robust, low-cost, integrated sensing suitable for complex robotic manipulation across varied environments, with strong potential for real-world deployment and multimodal robot learning.

Abstract

In this paper, we present the design and benchmark of an innovative sensor, ViTacTip, which fulfills the demand for advanced multi-modal sensing in a compact design. A notable feature of ViTacTip is its transparent skin, which incorporates a `see-through-skin' mechanism. This mechanism aims at capturing detailed object features upon contact, significantly improving both vision-based and proximity perception capabilities. In parallel, the biomimetic tips embedded in the sensor's skin are designed to amplify contact details, thus substantially augmenting tactile and derived force perception abilities. To demonstrate the multi-modal capabilities of ViTacTip, we developed a multi-task learning model that enables simultaneous recognition of hardness, material, and textures. To assess the functionality and validate the versatility of ViTacTip, we conducted extensive benchmarking experiments, including object recognition, contact point detection, pose regression, and grating identification. To facilitate seamless switching between various sensing modalities, we employed a Generative Adversarial Network (GAN)-based approach. This method enhances the applicability of the ViTacTip sensor across diverse environments by enabling cross-modality interpretation.

Figures (11)

• Figure 1: Overview of the four perception capabilities of ViTacTip, including two principle modalities: visual and tactile sensing, as well as two derived modalities: proximity and force sensing.
• Figure 2: (a) Architecture of the ViTacTip Sensor: An exploded view illustrating the sub-components, a side view of the assembled ViTacTip, and a detailed illustration of the sensor's tips and markers. (b) Demonstration of ViTacTip's proximity and vision perception capabilities. (c) Sketch of ViTacTip’s working principles: A: Projections of tactile deformations mapped by the displacement (${\Delta}x$) of black markers on the sensor tips. B/C: Visual feature projections captured through the sensor's transparent skin. D: Internal lighting passes through the skin, illuminating nearby objects. The sketch also compares the sensor's performance with and without a pin-like marker design.
• Figure 3: The design and fabrication schematic of the ViTacTip: (a) Entire Model Design: Exploded view of subcomponents. (b) Fabrication Process: The mounting bases and outer skin are 3D printed. An acrylic lens is glued to the base, and the gel is prepared and injected lepora2022digitaclin2022tactile. (c) Assembly Process: After fabricating the ViTacTip outer skin, it is assembled with the camera unit, illumination unit, and mounting base to construct the complete system.
• Figure 4: (a) The three stages of the ViTacTip sensing process and the illustration of the proximity perception mechanism. (b) Curve showing the relationship between distance (0-18 mm) and SSIM values of images obtained from ViTacTip, using a human finger database as an example. Points 'A', 'B', and 'C' correspond to the three stages illustrated in (a). The blue and red lines indicate the thresholds for segmenting the stages between A and B, and B and C, respectively. (c) GPR-based distance estimation using a database involves approaching human finger with ViTactip. (d) Examples of images captured during proximity perception between ViTacTip and three cubes with different textures. (e) The mean average errors (MAE) of ViTacTip in force estimation ($F_x$, $F_y$, $F_z$). Black dots represent the results of the trained model on the test dataset, while the red line represents the smoothed predictions.
• Figure 5: (a) Samples for the object recognition task: 21 objects with different shapes, as detailed in gomes2021generation. Real imaging from ViTacTip allows recognition of object localization and contour, with marker distribution adapting to the contact shape. (b) High-resolution perception images from ViTacTip and TacTip obtained by interacting with typical objects, illustrating challenges in shape differentiation using tactile information alone without visual support. (c) Experimental setup: a desktop Dobot robotic arm (MG400) for data collection lepora2022digitac, examples of texture samples used in experiments, and a schematic of hybrid visual-tactile sensing evaluation. Covering the elastomer with fabric requires ViTacTip to perceive both the elastomer's hardness and the fabric's texture upon contact.