Representing Data in Robotic Tactile Perception -- A Review

TL;DR

This paper reviews how data representation of tactile information shapes robotic perception from sensing to action. It identifies six common representations and analyzes how hardware morphology, taxel distribution, and high-level computation interact to determine what can be encoded. The authors propose guidelines for selecting representations based on hardware, encoded tactile information, and task requirements, and discuss multi-modal and cross-modal integration. They also highlight gaps such as calibration challenges for large-area sensors, the need for multimodal middleware, and potential of learned representations and Transformers to generalize across embodiments. The work argues for moving toward embodied, predictive tactile representations to improve safety and manipulation in unstructured environments.

Abstract

Robotic tactile perception is a complex process involving several computational steps performed at different levels. Tactile information is shaped by the interplay of robot actions, the mechanical properties of its body, and the software that processes the data. In this respect, high-level computation, required to process and extract information, is commonly performed by adapting existing techniques from other domains, such as computer vision, which expects input data to be properly structured. Therefore, it is necessary to transform tactile sensor data to match a specific data structure. This operation directly affects the tactile information encoded and, as a consequence, the task execution. This survey aims to address this specific aspect of the tactile perception pipeline, namely Data Representation. The paper first clearly defines its contributions to the perception pipeline and then reviews how previous studies have dealt with the problem of representing tactile information, investigating the relationships among hardware, representations, and high-level computation methods. The analysis has led to the identification of six structures commonly used in the literature to represent data. The manuscript provides discussions and guidelines for properly selecting a representation depending on operating conditions, including the available hardware, the tactile information required to be encoded, and the task at hand.

Figures (10)

• Figure 1: General overview of the operations implementing a tactile perception pipeline. This review focuses on the Data Representation block highlighted in bold. Tactile perception is action-conditioned, thus creating the circular dependency among the five different operations shown in the figure. As an example, the image shows a possible content for each block in a task of tactile-based exploration using a robotic hand embedding tactile receptors. The Data Acquisition block collects measurements from distributed sensors (previously filtered by the morphology of the body and robot actions) providing raw data as an output. The Data Representation block transforms the input to provide structured data that can be further processed at higher levels. The image shows some of the data structures used to represent data. In this example of tactile exploration the point cloud, representing the position of each receptor in space, is used. Red dots correspond to receptors in contact with the mug. Structured data is then further elaborated to extract features of interest from the contact, or as shown in this case, to merge multi-contact information and create a point cloud representation of the object. The inference step is related to high-level computation performing decision making. In this example, a data-driven model assigning a probability to each label, decides where to move at the next stage to increase the classification accuracy. Finally, the robot applies the Action, which affects the perceived information. In the example, the hand is moved to a target position to grasp the object with a desired force, closing the loop.
• Figure 2: A robot equipped with distributed tactile sensors represented as triangles. The image shows a high-level overview of the three layers composing a tactile sensor. The compliant layer is made of a material that deforms when a force is applied to it. The deformation is then captured by transducers. The force measurements are then collected, organized and transmitted as raw data.
• Figure 3: Two different ways exploited to rearrange raw data into a matrix structure. Yellow pads correspond to taxels distributed all over the hand. Using this representation methods can be used to process data acquired in different contact positions or to extract time-dependent features. (Left) Raw data acquired from different contacts are stacked over the rows. (Right) The time series acquired from a single sensor is encoded as a row in the matrix.
• Figure 4: A map representation conceptually similar to the one proposed by hoffmann_2018. Groups of taxels are stimulated to learn their spatial adjacency. Each hexagon in the map corresponds to a cluster position in 2D, with an associated value derived from the taxels' responses. The layout preserves proximity relations among clusters and across different body parts.
• Figure 5: Steps to encode tactile data into point clouds, meshes and images starting from a non-regular and non-planar arrangement of the taxels. If taxels are spatially calibrated, their distribution in the space can be represented as a point cloud that can be further processed to obtain a mesh. The same mesh can then be flattened to lie on a plane and resampled with a regular grid. The pixels' values in the image are computed by interpolation considering the measurements of the three adjacent taxels highlighted in orange.