Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Marker or Markerless? Mode-Switchable Optical Tactile Sensing for Diverse Robot Tasks

Ni Ou, Zhuo Chen, Shan Luo

TL;DR

The paper tackles the challenge of balancing marker-based manipulation and markerless perception in optical tactile sensors by introducing a software-only, mode-switchable framework. It combines a diffusion-based TacDiff inpainting method for marker-to-markerless transitions with a sparsely supervised markerless-marker regressor to enable bidirectional switching without additional hardware. Extensive upstream and downstream experiments demonstrate improved tactile image quality, more accurate marker motion estimation, and enhanced perception and manipulation performance, including higher texture classification accuracy, better contact-area segmentation, and robust slip detection. The approach enables a single tactile sensor to operate effectively across perception and manipulation tasks, with demonstrated generalization to unseen indenters and depths, and opens avenues for sim-to-real extensions and more dexterous tactile control.

Abstract

Optical tactile sensors play a pivotal role in robot perception and manipulation tasks. The membrane of these sensors can be painted with markers or remain markerless, enabling them to function in either marker or markerless mode. However, this uni-modal selection means the sensor is only suitable for either manipulation or perception tasks. While markers are vital for manipulation, they can also obstruct the camera, thereby impeding perception. The dilemma of selecting between marker and markerless modes presents a significant obstacle. To address this issue, we propose a novel mode-switchable optical tactile sensing approach that facilitates transitions between the two modes. The marker-to-markerless transition is achieved through a generative model, whereas its inverse transition is realized using a sparsely supervised regressive model. Our approach allows a single-mode optical sensor to operate effectively in both marker and markerless modes without the need for additional hardware, making it well-suited for both perception and manipulation tasks. Extensive experiments validate the effectiveness of our method. For perception tasks, our approach decreases the number of categories that include misclassified samples by 2 and improves contact area segmentation IoU by 3.53%. For manipulation tasks, our method attains a high success rate of 92.59% in slip detection. Code, dataset and demo videos are available at the project website: https://gitouni.github.io/Marker-Markerless-Transition/

Marker or Markerless? Mode-Switchable Optical Tactile Sensing for Diverse Robot Tasks

TL;DR

The paper tackles the challenge of balancing marker-based manipulation and markerless perception in optical tactile sensors by introducing a software-only, mode-switchable framework. It combines a diffusion-based TacDiff inpainting method for marker-to-markerless transitions with a sparsely supervised markerless-marker regressor to enable bidirectional switching without additional hardware. Extensive upstream and downstream experiments demonstrate improved tactile image quality, more accurate marker motion estimation, and enhanced perception and manipulation performance, including higher texture classification accuracy, better contact-area segmentation, and robust slip detection. The approach enables a single tactile sensor to operate effectively across perception and manipulation tasks, with demonstrated generalization to unseen indenters and depths, and opens avenues for sim-to-real extensions and more dexterous tactile control.

Abstract

Optical tactile sensors play a pivotal role in robot perception and manipulation tasks. The membrane of these sensors can be painted with markers or remain markerless, enabling them to function in either marker or markerless mode. However, this uni-modal selection means the sensor is only suitable for either manipulation or perception tasks. While markers are vital for manipulation, they can also obstruct the camera, thereby impeding perception. The dilemma of selecting between marker and markerless modes presents a significant obstacle. To address this issue, we propose a novel mode-switchable optical tactile sensing approach that facilitates transitions between the two modes. The marker-to-markerless transition is achieved through a generative model, whereas its inverse transition is realized using a sparsely supervised regressive model. Our approach allows a single-mode optical sensor to operate effectively in both marker and markerless modes without the need for additional hardware, making it well-suited for both perception and manipulation tasks. Extensive experiments validate the effectiveness of our method. For perception tasks, our approach decreases the number of categories that include misclassified samples by 2 and improves contact area segmentation IoU by 3.53%. For manipulation tasks, our method attains a high success rate of 92.59% in slip detection. Code, dataset and demo videos are available at the project website: https://gitouni.github.io/Marker-Markerless-Transition/
Paper Structure (18 sections, 6 equations, 10 figures, 4 tables, 1 algorithm)

This paper contains 18 sections, 6 equations, 10 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Optical Tactile Sensors
  4. Generative Models for Optical Tactile Sensing
  5. Method
  6. Overview
  7. Marker-Markerless Transition
  8. Markerless-Marker Transition
  9. Experiments
  10. Dataset Description
  11. Upstream Experiments
  12. Marker-markerless
  13. Markerless-marker
  14. Downstream Experiments
  15. Classification and Segmentation
  16. ...and 3 more sections

Figures (10)

  • Figure 1: Bidirectional transitions between marker and markerless modes. Marker-markerless transition (top): black markers are replaced with photo-realistic pixels; markerless-marker transition (bottom): pseudo marker motions (yellow arrows) are generated from a markerless tactile image.
  • Figure 2: Pipelines of the two transitions in our mode-switchable approach. The marker-markerless transition (left) is implemented by an inpainting method, with the tactile image with markers $I_w$ and the mask of markers $I_M$ as inputs. The markerless-marker transition (right) is realized by a encoder-decoder network that generates pseudo marker motions $\hat{M}_v$ from a markerless tactile image $I_{w/o}$. The selective output module retrieves features of sparse pixels to output 2D marker motions. Yellow arrows show the orientation and magnitude of marker motions.
  • Figure 3: Patch-based Merging. The resulting image is obtained by merging six small patches that are separately predicted by the inpainting model. The regions of patches in the merged image are annotated with white dashed rectangles, which overlap each other.
  • Figure 4: Training data acquisition for the marker-markerless transition. $I_M$ is extracted from $I_w$ and then placed onto $I_{w/o}$ to form the new tactile image with markers $I_{w/o}+I_M$. $I_{w/o}+I_M$ serves as input while $I_{w/o}$ serves as the ground-truth output.
  • Figure 5: Marker-offset strategy for training TacDiff if Sensor WO is not available. The TELEA inpainting algorithm TELEA is applied to generate $\hat{I}_{w/o}$. $I_M$ is translated by an offset to get $I'_M$. $\hat{I}_{w/o}$ and $I'_M$ are jointly cropped to obtain a pair of cropped patches $C(\hat{I}_{w/o})$ and $C(I'_M)$ for training TacDiff.
  • ...and 5 more figures