GelSight FlexiRay: Breaking Planar Limits by Harnessing Large Deformations for Flexible,Full-Coverage Multimodal Sensing

TL;DR

The paper tackles the challenge of embedding high-resolution tactile sensing into soft, deformable grippers without sacrificing compliance. It introduces GelSight FlexiRay, a flexible visual-tactile sensor integrated with Fin Ray soft grippers and a CMA-ES-optimized multi-mirror optical system to achieve full-coverage sensing under large deformations. The work delivers multimodal perception including force, position, texture, temperature, and slip, validated by quantitative results such as a force accuracy of 0.14 N and proprioceptive positioning accuracy of 0.19 mm, along with fivefold greater deformation capacity than comparable compliant VTS. The approach enables robust grasping, texture-based object recognition, and safe human-robot interaction, showing strong potential for scalable soft-robotic sensing in real-world manipulation tasks.

Abstract

The integration of tactile sensing into compliant soft robotic grippers offers a compelling pathway toward advanced robotic grasping and safer human-robot interactions. Visual-tactile sensors realize high-resolution, large-area tactile perception with affordable cameras. However, conventional visual-tactile sensors rely heavily on rigid forms, sacrificing finger compliance and sensing regions to achieve localized tactile feedback. Enabling seamless, large-area tactile sensing in soft grippers remains challenging, as deformations inherent to soft structures can obstruct the optical path and restrict the camera's field of view. To address these, we present Gelsight FlexiRay, a multimodal visual-tactile sensor designed for safe and compliant interactions with substantial structural deformation through integration with Finray Effect grippers. First, we adopt a multi-mirror configuration, which is systematically modeled and optimized based on the physical force-deformation characteristics of FRE grippers. Second, we enhanced Gelsight FlexiRay with human-like multimodal perception, including contact force and location, proprioception, temperature, texture, and slippage. Experiments demonstrate Gelsight FlexiRay's robust tactile performance across diverse deformation states, achieving a force measurement accuracy of 0.14 N and proprioceptive positioning accuracy of 0.19 mm. Compared with state of art compliant VTS, the FlexiRay demonstrates 5 times larger structural deformation under the same loads. Its expanded sensing area and ability to distinguish contact information and execute grasping and classification tasks highlights its potential for versatile, large-area multimodal tactile sensing integration within soft robotic systems. This work establishes a foundation for flexible, high-resolution tactile sensing in compliant robotic applications.

Figures (8)

• Figure 1: Gelsight FlexiRay: A novel flexible multimodal visual-tactile sensor, integrated with Fin Ray structures, delivers high-resolution tactile sensing on contact force, texture, slip, temperature, and proprioception.
• Figure 2: Illustration of the integrated design and layout optimization of Gelsight FlexiRay. (A) Exploded view of Gelsight FlexiRay. (B) Exploded view of the tactile sensing pad. (C) 3D perspective of the multi-mirror layout inside Gelsight FlexiRay. (D) Schematic representation of optical system optimization principles in a 2D cross-sectional view. (E) Camera FOV coverage under different deformation conditions due to varying loads.
• Figure 3: Fabrication process of Gelsight FlexiRay. (A) Casting of the reflective layer. (B) Engraving markers with a laser. (C) Casting of the temperature sensing layer. (D) Casting of the soft elastic layer. (E) String the LED lights in series and mount them onto the TPU beam. (F) Casting of the PDMS support. (G) Assemble the remaining 3D-printed parts to form the gripper.
• Figure 4: Perception model architecture. (A) Real-time semantic segmentation model for segmenting the front beam skeleton, perception region, and contact region. (B) Regression model for sensing interaction normal force, position, and proprioceptive structure deformation within the perception region. (C) Marker color mapping model for temperature sensing in the contact region. (D) Tactile-based object classification model for texture perception and recognition.
• Figure 5: Force and proprioceptive perception experimental process and results. (A) Data collection platform. (B) Probe type of the load cell. (C) Accuracy analysis of force estimation. (D) Absolute error distribution of contact position prediction. (E) Box plot of positioning errors for joint nodes. (F) Continuous estimation of contact force and contact depth during dynamic contact processes.