Learning Precise, Contact-Rich Manipulation through Uncalibrated Tactile Skins

TL;DR

This work presents the Visuo-Skin (ViSk) framework, a simple approach that uses a transformer-based policy and treats skin sensor data as additional tokens alongside visual information that significantly outperforms both vision-only and optical tactile sensing based policies.

Abstract

While visuomotor policy learning has advanced robotic manipulation, precisely executing contact-rich tasks remains challenging due to the limitations of vision in reasoning about physical interactions. To address this, recent work has sought to integrate tactile sensing into policy learning. However, many existing approaches rely on optical tactile sensors that are either restricted to recognition tasks or require complex dimensionality reduction steps for policy learning. In this work, we explore learning policies with magnetic skin sensors, which are inherently low-dimensional, highly sensitive, and inexpensive to integrate with robotic platforms. To leverage these sensors effectively, we present the Visuo-Skin (ViSk) framework, a simple approach that uses a transformer-based policy and treats skin sensor data as additional tokens alongside visual information. Evaluated on four complex real-world tasks involving credit card swiping, plug insertion, USB insertion, and bookshelf retrieval, ViSk significantly outperforms both vision-only and optical tactile sensing based policies. Further analysis reveals that combining tactile and visual modalities enhances policy performance and spatial generalization, achieving an average improvement of 27.5% across tasks. https://visuoskin.github.io/

Figures (5)

• Figure 1: ViSk uses AnySkin with a simple transformer-based architecture to solve precise, contact-rich tasks.
• Figure 2: (left) Robot setup used for experiments in Section \ref{['sec:experiments']}; (right) ViSk policy architecture uses ResNet-18 he2016deep encoders for camera inputs and an MLP encoder for AnySkin input. An action token is appended to the encoded inputs before passing them through a transformer decoder, and the corresponding feature is used for action prediction by the action head.
• Figure 3: Close-up views of ViSk rollouts for the four tasks: Plug Insertion, Card Swiping, USB Insertion and Book Retrieval
• Figure 4: Overhead view depicting variations in target object locations for training and evaluation for plug insertion, card swiping and USB insertion (left to right). The enclosing light green box denotes the extent of variation in the training data. Test locations for plug insertion and USB insertion are marked on the image. For the card swiping task, arrows denote test locations and orientations of the card machine used for evaluation. For the book retrieval task (not depicted here), the order of books is randomized for every training demonstration, and test configurations consist of orderings unseen in training data.
• Figure 5: We vary different parameters of the object used for collecting demonstrations to analyze the generalizability of ViSk policies for the four tasks: (top to bottom) Plug Insertion, USB Insertion, Card Swiping and Book Retrieval.