AllSight: A Low-Cost and High-Resolution Round Tactile Sensor with Zero-Shot Learning Capability

TL;DR

AllSight addresses the demand for high-resolution tactile sensing on round surfaces by introducing a thumb-sized, low-cost optical sensor with a 3D-printed shell and modular construction. The key idea is to jointly estimate the full contact state, including position, normal and tangential forces, and torsion, using a learning pipeline that leverages simulated data (via the TACTO simulator) for pretraining and real data for finetuning, enabling zero-shot transfer to newly fabricated sensors. The authors compare illumination strategies and the use of markers on the elastomer, finding that a RRRGGGBBB illumination with markers yields the best accuracy, while highlighting the effectiveness of sim-to-real pretraining to reduce data collection. The approach demonstrates a practical zero-shot capability, offering an open-source, ready-to-use model for new sensors, with potential impact on low-cost, reproducible tactile sensing for multi-finger robotic manipulation.

Abstract

Tactile sensing is a necessary capability for a robotic hand to perform fine manipulations and interact with the environment. Optical sensors are a promising solution for high-resolution contact estimation. Nevertheless, they are usually not easy to fabricate and require individual calibration in order to acquire sufficient accuracy. In this letter, we propose AllSight, an optical tactile sensor with a round 3D structure potentially designed for robotic in-hand manipulation tasks. AllSight is mostly 3D printed making it low-cost, modular, durable and in the size of a human thumb while with a large contact surface. We show the ability of AllSight to learn and estimate a full contact state, i.e., contact position, forces and torsion. With that, an experimental benchmark between various configurations of illumination and contact elastomers are provided. Furthermore, the robust design of AllSight provides it with a unique zero-shot capability such that a practitioner can fabricate the open-source design and have a ready-to-use state estimation model. A set of experiments demonstrates the accurate state estimation performance of AllSight.

Figures (10)

• Figure 1: Three AllSight sensors on the fingers of an OpenHand Model-O Ma2017YaleOP robotic hand. The sensors provide real-time tactile images for contact state estimations during the manipulation of an object. Surface deformations due to contact are marked with a circle at the bottom tactile images.
• Figure 2: Illustration of AllSight (1) assembled and (2) in an exploded view. (3) Images of the corresponding fabricated parts are seen including marked and clear elastomers. (4) Three different LED illumination configurations when (5) AllSight is in contact with a screw: (4a) white, (4b) RRRGGGBBB and (4c) RGBRGBRGB. Top and bottom rows of the camera view show elastomers with and without markers, respectively.
• Figure 3: Steps 1-3 depict the fabrication process of the elastomer including (1) mold printing, (2) molding and (3) reflective coating. Steps 4-5 show the fabrication of the clear rigid shell through (4) 3D printing with clear resin and (5) polishing. In step 6, the coated elastomer is glued onto the rigid shell.
• Figure 4: Tactile images during contact of AllSight with (left to right) a circuit board, a screw driver, a locking pliers, a screw and an embossed 'x'.
• Figure 5: The contact state is defined by the position $\mathbf{x}$ of contact with respect to the sensors coordinate frame, normal force $f_z$ at the contact, tangential forces $f_x$ and $f_y$ and the torsion $\tau$ about the normal axis.