Digitizing Touch with an Artificial Multimodal Fingertip

TL;DR

The results demonstrate the possibility of digitizing touch with superhuman performance and describe several conceptual and technological innovations to improve the digitization of touch that are embodied in an artificial finger-shaped sensor with advanced sensing capabilities.

Abstract

Touch is a crucial sensing modality that provides rich information about object properties and interactions with the physical environment. Humans and robots both benefit from using touch to perceive and interact with the surrounding environment (Johansson and Flanagan, 2009; Li et al., 2020; Calandra et al., 2017). However, no existing systems provide rich, multi-modal digital touch-sensing capabilities through a hemispherical compliant embodiment. Here, we describe several conceptual and technological innovations to improve the digitization of touch. These advances are embodied in an artificial finger-shaped sensor with advanced sensing capabilities. Significantly, this fingertip contains high-resolution sensors (~8.3 million taxels) that respond to omnidirectional touch, capture multi-modal signals, and use on-device artificial intelligence to process the data in real time. Evaluations show that the artificial fingertip can resolve spatial features as small as 7 um, sense normal and shear forces with a resolution of 1.01 mN and 1.27 mN, respectively, perceive vibrations up to 10 kHz, sense heat, and even sense odor. Furthermore, it embeds an on-device AI neural network accelerator that acts as a peripheral nervous system on a robot and mimics the reflex arc found in humans. These results demonstrate the possibility of digitizing touch with superhuman performance. The implications are profound, and we anticipate potential applications in robotics (industrial, medical, agricultural, and consumer-level), virtual reality and telepresence, prosthetics, and e-commerce. Toward digitizing touch at scale, we open-source a modular platform to facilitate future research on the nature of touch.

Figures (25)

• Figure 1: Digitizing touch with the artificial multimodal fingertip.
• Figure 2: a) We evaluated normal and shear forces in three separate regions of the sensor, from the tip toward the side. The dots and error bars show the median and 95-percentile of the error, respectively. The median error from the deep-learning model for the three regions is 1.01 mN, 1.09 mN, 1.41 mN for normal forces, and 1.27 mN, 1.48 mN, 1.64 mN for shear forces. b) We train a deep learning model to predict normal and shear forces from visuo-tactile image output. Normal and shear forces are predicted with a median error of 1.01 mN and 1.27 mN, respectively. Compared to alternative methods, predicting shear force requires the use of markers; however, with increased spatial resolution, far more features are extracted from the visuo-tactile image, which aids in shear force prediction. c) We evaluate spatial resolution by using a dual-pronged microindenter depressed into the artificial fingertip with varying widths. Visual validation and the inspection of the taxels’ profile intensity confirmed the ability to clearly distinguish features as small as 7 $\mu m$. d) We show two methods that create the artificial fingertip volume: internal structure (top row) and solid gel with immersion lens (bottom row). With an internal structure, illumination artifacts are visible, whereas, with a solid volume, the resulting image is far higher quality with fewer illumination artifacts. e) We simulate the effects of increasing the surface scattering along the internal reflective layer of the artificial fingertip. From left to right, machine polish of 1° to Lambertian scattering, we optimize for image contrast while constraining the background illumination uniformity.
• Figure 3: a). We demonstrate the ability to determine the volume of water in an opaque container by tapping with one finger and recording the response on the fingers in contact with the container. We further show how this modality is deconstructed into peak frequency analysis, independent of finger position, whereas decay time depends on finger placement. b) Spectrogram of the surface audio textures recorded for different objects. c) Using a variable heat source as a control, the artificial fingertip is sensitive to heat gradients. d) The artificial fingertip provides object state and identification of objects through local gas sensing, achieving a 91% accuracy.e) The accuracy of object classification through scent depends on the integration time as the artificial fingertip begins approaching the object. We show that 61% accuracy is reached within 6 seconds. f) Localization of finger placement on an object during movement transients with empty and full volumes of liquid. We measure the effects of transients during impulse, resonance during a static hold, and a static hold with no movement.
• Figure 4: Different actions, materials and states during touch digitization uniqueliy contirbute to excitations in multimodal signals. We show in a) the action of tapping a wooden spatula and then transitioning to a stiring motion. In b), we show a state change between a raw egg and a hard-boiled egg through discrete differences in the dynamics when an impulse motion is applied while the egg is in a hand grasp. Furthermore, in c) We deploy a complex scenario of scene understanding through touch digitization in which a kitchen appliance and utensil are used with four artificial fingers to capture how modalities change over the course of making scrambled eggs.
• Figure 5: a) We introduce an analog of the human reflex arc by quickly processing sensory input within the fingertip, directly controlling the actuators of a robot hand to retract in response to touching an object. b) Typical tactile processing and control paradigm transfer of sensory data to a remote computer for processing. This requires sufficient bandwidth and introduces communication latency. c) Local processing for mimicking the reflex arc with the system fingertip. Our system can use the on-device AI neural network accelerator for local processing to decrease overall latency between event and action. d) Mean and standard deviation of the event-to-action latency. The on-device local processing and control loop takes 1.2 ms compared to the traditional paradigm, which takes 2.5 ms on a Digit 360 and over 6 ms on a Digit. e) A comparison between normal and shear force sensitivities across devices. f) We compare the performance of our artificial fingertip sensor against existing sensors, and we show that our sensor delivers significant improvements (4X for spatial, 6X for normal forces, and 9X for shear forces) on all the metrics evaluated.