Proximity and Visuotactile Point Cloud Fusion for Contact Patches in Extreme Deformation

TL;DR

This work tackles the challenge of estimating contact patches under extreme membrane deformation by fusing proximity and visuotactile point clouds in a mechanics-independent fashion. By intersecting the proximity-derived and tactile-derived patches, the method robustly identifies contact regions without relying on complex membrane models, enabling real-time feedback for control and pose estimation. Across low to high strain states and diverse objects, the approach consistently achieves RMSEs well below a few millimeters, outperforming tactile-only, proximity-only, and first-principles mechanics baselines. The demonstrated demonstrations—variable stiffness membranes, wrinkling, closed-loop manipulation, and pose estimation—highlight practical impact for soft robotics where membrane mechanics are hard to model precisely.

Abstract

Visuotactile sensors are a popular tactile sensing strategy due to high-fidelity estimates of local object geometry. However, existing algorithms for processing raw sensor inputs to useful intermediate signals such as contact patches struggle in high-deformation regimes. This is due to physical constraints imposed by sensor hardware and small-deformation assumptions used by mechanics-based models. In this work, we propose a fusion algorithm for proximity and visuotactile point clouds for contact patch segmentation, entirely independent from membrane mechanics. This algorithm exploits the synchronous, high spatial resolution proximity and visuotactile modalities enabled by an extremely deformable, selectively transmissive soft membrane, which uses visible light for visuotactile sensing and infrared light for proximity depth. We evaluate our contact patch algorithm in low (10%), medium (60%), and high (100%+) strain states. We compare our method against three baselines: proximity-only, tactile-only, and a first principles mechanics model. Our approach outperforms all baselines with an average RMSE under 2.8 mm of the contact patch geometry across all strain ranges. We demonstrate our contact patch algorithm in four applications: varied stiffness membranes, torque and shear-induced wrinkling, closed loop control, and pose estimation.

Figures (7)

• Figure 1: We propose a mechanics-independent algorithm for estimating contact patches in challenging deformation regimes. We leverage a selectively transmissive soft membrane to provide simultaneous tactile and proximity point clouds. By fusing these two modalities through simply computing the intersection between the two point clouds, we obtain a high quality estimate of the contact patch. Because of our mechanics-independent approach, we avoid the non-linearities that many existing contact patches algorithm struggle with.
• Figure 2: The ground truth method for contact patch geometry exploits heat transfer between the object and sensor membrane at the contact interface. The object is painted with thermochromic paint that changes color when it comes into contact with the heated membrane.
• Figure 3: Method used to calculate membrane strain. A) Uniform grid of dots with known spacing is patterned on the membrane. B) Object (octopus, not shown) is pressed into the membrane and the RGB-D point cloud is captured. C) The centroids of the dots from the RGB-D point cloud are isolated. D) The distances between the centroids are measured to calculate strain.
• Figure 4: Visualizations of the estimated contact patch across low, medium, and high strain states of the membrane. Our proposed algorithm, Fusion, fuses the visuotactile and proximity modalities of the sensor enabled by the selectively transmissive membrane. The baselines for comparison are Proximity Only, Tactile Only, and Mechanics Model. The ground truth is a color-segmented 3D scan of the object, where pink designates contact.
• Figure 5: We demonstrate the proposed fusion algorithm for contact patch estimation in two applications with complex membrane mechanics: A. a varied stiffness membrane and, B. membrane wrinkles, or out-of-plane deformations.