PneuGelSight: Soft Robotic Vision-Based Proprioception and Tactile Sensing

TL;DR

This work addresses sensing for soft robots by developing PneuGelSight, a vision-based sensor embedded in a soft finger that enables simultaneous high-resolution proprioception and tactile surface reconstruction. It combines a sim-to-real pipeline with a shape-prior autoencoder and a PoseNet-like network to achieve zero-shot transfer, and a tactile reconstruction pipeline that uses region proposals, background estimation, and Poisson integration to recover 3D contact geometry. The approach delivers real-time proprioception (inference $<0.05$ s) and sub-millimeter tactile fidelity (e.g., Chamfer $\approx 0.18$ mm) across diverse contacts, validated on a UR5e platform and demonstrated in a real-world avocado grasp. This compact, easily implementable sensorization framework advances soft-robot perception for manipulation in unstructured environments by unifying high-resolution proprioception and tactile sensing through a single onboard camera and a physics-informed simulation design loop.

Abstract

Soft pneumatic robot manipulators are popular in industrial and human-interactive applications due to their compliance and flexibility. However, deploying them in real-world scenarios requires advanced sensing for tactile feedback and proprioception. Our work presents a novel vision-based approach for sensorizing soft robots. We demonstrate our approach on PneuGelSight, a pioneering pneumatic manipulator featuring high-resolution proprioception and tactile sensing via an embedded camera. To optimize the sensor's performance, we introduce a comprehensive pipeline that accurately simulates its optical and dynamic properties, facilitating a zero-shot knowledge transition from simulation to real-world applications. PneuGelSight and our sim-to-real pipeline provide a novel, easily implementable, and robust sensing methodology for soft robots, paving the way for the development of more advanced soft robots with enhanced sensory capabilities.

Figures (12)

• Figure 1: Design and Sensing Pipeline of PneuGelSight.(A) PneuGelSight gripper (left) grasping an object, with corresponding camera-captured images (middle) and high-resolution proprioceptive sensing result (top right) and tactile reconstruction of surface contact geometry (bottom right). (B) Mechanical design of the PneuGelSight sensor. The sensor integrates a deformable silicone layer, a reflective surface, and internal optical fibers for illumination. Upon contact, deformation alters the internal light pattern, which is captured by the embedded camera. (C) Data processing pipeline for sensing. A dual-branch network processes the captured image by extracting contour features for global shape reconstruction and color cues for local contact geometry.
• Figure 2: Optimization for Optical Design. Comparison of simulated images with various optical designs and real images under different bending angles, along with their corresponding variance scores. The design with the highest variance is selected for real-world fabrication.
• Figure 3: Overview of FEM scenes in the dataset, with the blue arrow indicating the movement direction of the external object.
• Figure 4: Proprioception Pipeline.(A) Pre-training the auto-encoder to reconstruct robot shape points cloud. (B) The architecture of proprioception network (ProprioNet). The feature extracted from the shape reference is modified by image feature in the latent space, and then mapped to the deformed point clouds.
• Figure 5: Overview of the tactile sensing network. Step 1, The raw image captured by the embedded camera is processed through region proposal and background subtraction to extract a color difference map highlighting the contact area. Step 2, The identified region is then fed into the 3D mesh reconstruction network to estimate the contact surface geometry.