GelFusion: Enhancing Robotic Manipulation under Visual Constraints via Visuotactile Fusion

TL;DR

GelFusion tackles robust robotic manipulation under vision-limited conditions by integrating high-resolution visuotactile feedback from GelSight sensors with vision through a cross-attention fusion mechanism within a diffusion-policy framework. It introduces a dual-channel tactile representation capturing static texture/geometry and dynamic interaction events, fused to visual features via a vision-led cross-attention module. Empirical results across surface wiping, peg insertion, and fragile object pick demonstrate improved success rates over vision-only baselines and ablations, especially in contact-rich and occluded scenarios. This approach suggests that visuotactile fusion can significantly enhance policy learning in unstructured settings, enabling more reliable manipulation when visual information is incomplete or misleading.

Abstract

Visuotactile sensing offers rich contact information that can help mitigate performance bottlenecks in imitation learning, particularly under vision-limited conditions, such as ambiguous visual cues or occlusions. Effectively fusing visual and visuotactile modalities, however, presents ongoing challenges. We introduce GelFusion, a framework designed to enhance policies by integrating visuotactile feedback, specifically from high-resolution GelSight sensors. GelFusion using a vision-dominated cross-attention fusion mechanism incorporates visuotactile information into policy learning. To better provide rich contact information, the framework's core component is our dual-channel visuotactile feature representation, simultaneously leveraging both texture-geometric and dynamic interaction features. We evaluated GelFusion on three contact-rich tasks: surface wiping, peg insertion, and fragile object pick-and-place. Outperforming baselines, GelFusion shows the value of its structure in improving the success rate of policy learning.

Figures (12)

• Figure 1: Visionlimited conditions for policy learning. More specifically, (a) ambiguous visual cues specifically refers to situations where, when the field of view is restricted, it is difficult to obtain complete state estimation based solely on visual information. An example is the scene degradation that occurs when trying to erase a whiteboard, making depth difficult to estimate. (b) occlusion frequently occurs in operations involving assembly or tool use, often when the gripped object blocks the details required for critical steps.
• Figure 2: Visuotactile sensor dual-channel representation. (a) The self-made, low-cost Gelsight sensor used for data acquisition. (b) A single static frame provides object properties, such as texture and local geometric profile. (c) Temporal information across frames captures features of dynamic interaction, visualized here via a binary difference image between adjacent frames. (d) Analyzing temporal changes, specifically the variance of the binarized difference over time, explicitly reveals events like contact initiation and cessation, facilitating the learning of policy-relevant features and perception of subtle changes.
• Figure 3: Network Architecture.
• Figure 4: Vision-led Cross-attention Fusion.
• Figure 5: Wiping Evaluation. (a) Test scenarios used to evaluate policy generalization, including variations in wiping area, line shape, and initial height compared to training conditions. (b) Typical failure cases observed: the eraser floating above the table (Float) and the gripper overpressing the eraser (Overpressed). (c) and (d) show experimental results, comparing the success rate of our method ('Ours') against ablation baselines and variants.