PolyTouch: A Robust Multi-Modal Tactile Sensor for Contact-rich Manipulation Using Tactile-Diffusion Policies

TL;DR

Robotic manipulation in unstructured home environments is hindered by occlusions and complex contact interactions when relying on vision and proprioception alone. PolyTouch integrates camera-based tactile sensing, acoustic sensing, and peripheral vision into a compact finger and introduces a tactile-diffusion policy that fuses these modalities with vision and proprioception to learn robust contact-aware control. Across durability tests and four bimanual manipulation tasks, tactile-inclusive policies outperform state-of-the-art visuomotor baselines, demonstrating improved robustness and data efficiency, while PolyTouch emphasizes durability and manufacturability for large-scale policy learning. This work highlights the practical potential of multi-modal tactile sensing to accelerate the development of reliable domestic robots with versatile manipulation capabilities.

Abstract

Achieving robust dexterous manipulation in unstructured domestic environments remains a significant challenge in robotics. Even with state-of-the-art robot learning methods, haptic-oblivious control strategies (i.e. those relying only on external vision and/or proprioception) often fall short due to occlusions, visual complexities, and the need for precise contact interaction control. To address these limitations, we introduce PolyTouch, a novel robot finger that integrates camera-based tactile sensing, acoustic sensing, and peripheral visual sensing into a single design that is compact and durable. PolyTouch provides high-resolution tactile feedback across multiple temporal scales, which is essential for efficiently learning complex manipulation tasks. Experiments demonstrate an at least 20-fold increase in lifespan over commercial tactile sensors, with a design that is both easy to manufacture and scalable. We then use this multi-modal tactile feedback along with visuo-proprioceptive observations to synthesize a tactile-diffusion policy from human demonstrations; the resulting contact-aware control policy significantly outperforms haptic-oblivious policies in multiple contact-aware manipulation policies. This paper highlights how effectively integrating multi-modal contact sensing can hasten the development of effective contact-aware manipulation policies, paving the way for more reliable and versatile domestic robots. More information can be found at https://polytouch.alanz.info/

Figures (7)

• Figure 1: (a) PolyTouch is a robot finger that combines tactile, acoustic, and peripheral vision sensing. (b) We design 4 common bimanual tasks to evaluate tactile-diffusion policies: egg serving, fruit sorting, egg cracking, and wrench insertion.
• Figure 2: Drawings and main components of PolyTouch
• Figure 3: Explosion view of PolyTouch and its estimated cost. PolyTouch's BoM is mostly comprised of easily accessible materials, and its construction does not require specialized equipment.
• Figure 4: Optical simulation of PolyTouch. The light reflected by the sensing surface is redistributed by the curved mirror before reaching the camera. The camera sensor is 4:3 with a diagonal field of view of 120 degrees.
• Figure 5: The tactile-diffusion policy network. All sensing modalities coming from each of the two arms are encoded with pre-trained feature extractors (except proprioception which is encoded by a MLP from scratch), then combined by a mixture of cross attentions and concatenations, before passed into a diffusion head for action prediction.