Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Low Fidelity Visuo-Tactile Pretraining Improves Vision-Only Manipulation Performance

Selam Gano, Abraham George, Amir Barati Farimani

TL;DR

This work tackles the fragility and cost of tactile sensing by leveraging visuo-tactile pre-training with a low-cost BeadSight sensor, then discarding tactile data during inference to achieve robust vision-only manipulation. By pre-training tactile and visual encoders jointly and evaluating on USB cable plugging and a long-horizon drawer task, the authors show that BeadSight-driven pre-training can yield substantial gains in vision-only performance (up to 65% in the USB task) and that pre-training across similar, dissimilar, or identical tasks generally improves downstream outcomes. Key contributions include demonstrating the effectiveness of a low-fidelity tactile sensor for visuo-tactile pre-training, highlighting the benefits of freezing the tactile encoder to mitigate distribution drift, and establishing that visuo-tactile pre-training can enhance vision-only policies across tasks. The findings suggest a scalable pathway to improve real-world manipulation with inexpensive tactile sensing by leveraging pre-trained visuo-tactile encoders and careful ablations.

Abstract

Tactile perception is essential for real-world manipulation tasks, yet the high cost and fragility of tactile sensors can limit their practicality. In this work, we explore BeadSight (a low-cost, open-source tactile sensor) alongside a tactile pre-training approach, an alternative method to precise, pre-calibrated sensors. By pre-training with the tactile sensor and then disabling it during downstream tasks, we aim to enhance robustness and reduce costs in manipulation systems. We investigate whether tactile pre-training, even with a low-fidelity sensor like BeadSight, can improve the performance of an imitation learning agent on complex manipulation tasks. Through visuo-tactile pre-training on both similar and dissimilar tasks, we analyze its impact on a longer-horizon downstream task. Our experiments show that visuo-tactile pre-training improved performance on a USB cable plugging task by up to 65% with vision-only inference. Additionally, on a longer-horizon drawer pick-and-place task, pre-training--whether on a similar, dissimilar, or identical task--consistently improved performance, highlighting the potential for a large-scale visuo-tactile pre-trained encoder.

Low Fidelity Visuo-Tactile Pretraining Improves Vision-Only Manipulation Performance

TL;DR

This work tackles the fragility and cost of tactile sensing by leveraging visuo-tactile pre-training with a low-cost BeadSight sensor, then discarding tactile data during inference to achieve robust vision-only manipulation. By pre-training tactile and visual encoders jointly and evaluating on USB cable plugging and a long-horizon drawer task, the authors show that BeadSight-driven pre-training can yield substantial gains in vision-only performance (up to 65% in the USB task) and that pre-training across similar, dissimilar, or identical tasks generally improves downstream outcomes. Key contributions include demonstrating the effectiveness of a low-fidelity tactile sensor for visuo-tactile pre-training, highlighting the benefits of freezing the tactile encoder to mitigate distribution drift, and establishing that visuo-tactile pre-training can enhance vision-only policies across tasks. The findings suggest a scalable pathway to improve real-world manipulation with inexpensive tactile sensing by leveraging pre-trained visuo-tactile encoders and careful ablations.

Abstract

Tactile perception is essential for real-world manipulation tasks, yet the high cost and fragility of tactile sensors can limit their practicality. In this work, we explore BeadSight (a low-cost, open-source tactile sensor) alongside a tactile pre-training approach, an alternative method to precise, pre-calibrated sensors. By pre-training with the tactile sensor and then disabling it during downstream tasks, we aim to enhance robustness and reduce costs in manipulation systems. We investigate whether tactile pre-training, even with a low-fidelity sensor like BeadSight, can improve the performance of an imitation learning agent on complex manipulation tasks. Through visuo-tactile pre-training on both similar and dissimilar tasks, we analyze its impact on a longer-horizon downstream task. Our experiments show that visuo-tactile pre-training improved performance on a USB cable plugging task by up to 65% with vision-only inference. Additionally, on a longer-horizon drawer pick-and-place task, pre-training--whether on a similar, dissimilar, or identical task--consistently improved performance, highlighting the potential for a large-scale visuo-tactile pre-trained encoder.
Paper Structure (19 sections, 4 equations, 6 figures, 4 tables)

This paper contains 19 sections, 4 equations, 6 figures, 4 tables.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND AND RELATED WORK
  3. Tactile Sensing
  4. Tactile Sensing and Manipulation Control Policies
  5. Imitation Learning Methods
  6. Diffusion Policy
  7. Action Chunking Transformers (ACT)
  8. Contrastive Pre-training
  9. METHODS
  10. BeadSight Hardware
  11. Data Collection
  12. Tactile Encoder
  13. Contrastive pre-training
  14. Diffusion Policy
  15. Action Chunking Transformers
  16. ...and 4 more sections

Figures (6)

  • Figure 1: A diagram of the flow of information through tactile pre-training and inference. During contrastive pre-training, primed tactile and visual encoders are produced. The encoders are further refined on the downstream task. Then, the tactile encoder is discarded for vision-only inference without the need for a tactile sensor.
  • Figure 2: BeadSight components and construction, including (from left to right) the camera, housing, lid frame, acrylic panel, and bead sac on a light-blocking backing
  • Figure 3: The embedded camera's view of the BeadSight hydro-gel bead sac, relaxed (1) and pressed (2), after down-scaling and correcting fish eye distortion.
  • Figure 4: USB cable plugging experimental scene, with a D415 wrist-mounted camera (not visible) positioned behind the robot gripper.
  • Figure 5: Drawer placement task experimental scene, including a side view of the BeadSight Sensor attached to the end effector.
  • ...and 1 more figures