ThinTact:Thin Vision-Based Tactile Sensor by Lensless Imaging

TL;DR

ThinTact introduces a thin lensless vision-based tactile sensor with a sensing field over 200 mm^2 and thickness under 10 mm, enabled by an amplitude-mask and a non-iterative, DCT-based reconstruction algorithm. A GA-driven mask optimization and calibration of system matrices improve reconstruction quality for close-up imaging, while Real2Sim enhances depth accuracy for tactile sensing. The approach achieves real-time performance (sub-2 ms per frame on GPU) and demonstrates capabilities in texture recognition and delicate object manipulation within constrained spaces. The work offers a practical path toward ultra-thin, high-resolution tactile sensing suitable for compact grippers and dexterous robots, with future directions including further thickness reduction and faster processing.

Abstract

Vision-based tactile sensors have drawn increasing interest in the robotics community. However, traditional lens-based designs impose minimum thickness constraints on these sensors, limiting their applicability in space-restricted settings. In this paper, we propose ThinTact, a novel lensless vision-based tactile sensor with a sensing field of over 200 mm2 and a thickness of less than 10 mm.ThinTact utilizes the mask-based lensless imaging technique to map the contact information to CMOS signals. To ensure real-time tactile sensing, we propose a real-time lensless reconstruction algorithm that leverages a frequency-spatial-domain joint filter based on discrete cosine transform (DCT). This algorithm achieves computation significantly faster than existing optimization-based methods. Additionally, to improve the sensing quality, we develop a mask optimization method based on the generic algorithm and the corresponding system matrix calibration algorithm.We evaluate the performance of our proposed lensless reconstruction and tactile sensing through qualitative and quantitative experiments. Furthermore, we demonstrate ThinTact's practical applicability in diverse applications, including texture recognition and contact-rich object manipulation. The paper will appear in the IEEE Transactions on Robotics: https://ieeexplore.ieee.org/document/10842357. Video: https://youtu.be/YrOO9BDMAHo

Figures (19)

• Figure 1: In this work, we propose ThinTact, a thin vision-based tactile sensor with a thickness of less than 10 mm. To overcome the thickness constraint of the lens system, we utilize the amplitude-mask-based lensless imaging technique to translate the contact information into CMOS signals. We first reconstruct a clear image from the CMOS signal, and then compute the contact geometry and marker displacements. The high sensitivity and thin profile of ThinTact enables many applications, including texture recognition, delicate object grasping and object manipulation.
• Figure 2: Workflow of ThinTact. On the left of this figure is the schematic of ThinTact. It utilizes a separable amplitude mask for imaging. To address the challenging problems associated with integrating lensless imaging into tactile sensing, we have developed a non-iterative real-time reconstruction algorithm, a mask optimization method, and a corresponding calibration algorithm. The reconstructed images allow us to determine contact geometry and marker displacements, which helps robot manipulation.
• Figure 3: In the proposed tactile sensor, the scene is close to the CMOS. (a) Close-up imaging situation. In this case, the scene-to-CMOS distance is smaller than the sensor size so that the PSF only covers a small area. (b) The resulting PSF. In the T2S model, the PSF can be decomposed into two terms: (c) the open component and (d) the coding component. (e) System matrix $\bm{P}_\text{o}$. (f) System matrix $\bm{P}_\text{c}$.
• Figure 4: The pipeline of the proposed non-iterative reconstruction algorithm.
• Figure 5: Analysis of system matrices. (a) Randomized system matrix generated in khan2020flatnet. In this work, we generate system matrices that are suitable for close-up imaging scenarios. (b) Analysis of the system matrix. The vector on the left is used to form the mask and the corresponding system matrix is on the right. They are assumed to be aligned at their centers. The matrix has several key parameters: the slope $k_\text{c}$ of the center line of the striped area (the purple dashed line), the slope $k_\text{stripe}$ of the direction of the stripes (the red solid line), the width of the striped area $d_\text{stripe}$, and the effective length $l_\text{mask}$ in the vector (marked by the yellow dashed lines).