A Modularized Design Approach for GelSight Family of Vision-based Tactile Sensors

TL;DR

GelSight vision-based tactile sensors enable high-resolution contact geometry measurement, but adapting them to new robot hands requires laborious optical redesigns. The authors introduce a modular, parameterized design framework and a physics-based optical simulator, implemented as OptiSense Studio, to enable rapid forward and inverse sensor design with sim-to-real transfer. Four objective functions (RGB2Normal, NormDiff, AOAP, 2to3PW) quantify sensing performance, complemented by a component library and CMA-ES/grid-search optimization to explore design spaces efficiently. Through four case studies, the framework demonstrates strong sim-to-real alignment and substantial reductions in design time, while enabling new sensor geometries and illumination strategies. The approach also provides a foundation for task-driven and cross-platform optimization of vision-based tactile sensors.

Abstract

GelSight family of vision-based tactile sensors has proven to be effective for multiple robot perception and manipulation tasks. These sensors are based on an internal optical system and an embedded camera to capture the deformation of the soft sensor surface, inferring the high-resolution geometry of the objects in contact. However, customizing the sensors for different robot hands requires a tedious trial-and-error process to re-design the optical system. In this paper, we formulate the GelSight sensor design process as a systematic and objective-driven design problem and perform the design optimization with a physically accurate optical simulation. The method is based on modularizing and parameterizing the sensor's optical components and designing four generalizable objective functions to evaluate the sensor. We implement the method with an interactive and easy-to-use toolbox called OptiSense Studio. With the toolbox, non-sensor experts can quickly optimize their sensor design in both forward and inverse ways following our predefined modules and steps. We demonstrate our system with four different GelSight sensors by quickly optimizing their initial design in simulation and transferring it to the real sensors.

Figures (18)

• Figure 1: Framework for modularizing and parameterizing camera-based sensors: In (A), the user designs an initial CAD that contains the locations and shapes of the sensor components. (B) This CAD design is converted to a complete sensor design in OptiSense Studio. We automatically parameterize the optical design and optimize it using novel objective functions in our simulation-driven framework. (C) The final design can then be directly manufactured and tested in the real world.
• Figure 2: Various types of GelSight sensor designs (Pictures adapted from li2014localizationslipdetectionlambeta2020digittaylor2021gelslim3finrayliu2022gelsightbrandenroundsensordensetactv1tippur2023gelsight360sveltezhao2023gelsightgelfinger): Researchers have designed GelSight sensors with varied optical systems to fit robot fingers with either flat or curved surfaces. The performance of the sensors also varies and is affected by a number of parameters of the optical system. Our work aims to bridge the knowledge gap between expert sensor designers and novice users, thereby simplifying and expediting the sensor design process.
• Figure 3: Sensor design framework: Given the user shape input in A, we model the sensor design with multiple modules in simulation as shown in B. We then evaluate the sensor performance based on the simulated indentation test in C. This is then coupled with optimization methods to choose the optimal light module and optical coating material for the sensor design.
• Figure 4: Sensor modularization: This figure illustrates how a tactile sensor can be modularized into our proposed modules. These modules can then be used to create a digital design for further optimization.
• Figure 5: Shape parameterization: We use a cage (the bounding box of the mesh surface) to parameterize the component shape, shown in the left column as M1 representing mirrors in GelSight Svelte. The cage is set as the geometrical parent for the target surface mesh (child). Changing the shape of the cage consequently changes the mesh. The right column shows the user input of optimization boundary $\mathcal{C}_{\text{min}}$ and $\mathcal{C}_{\text{max}}$. We show the deformed surface, 2D profile, and the corresponding tactile image to qualitatively represent the change in tactile signal by changing M1 mirror element in the sensor. Therefore, shape optimization of M1 is critical to obtaining the best sensing performance.