ViTacGen: Robotic Pushing with Vision-to-Touch Generation

TL;DR

ViTacGen tackles the challenge of tactile-sensor dependence in robotic pushing by learning to generate tactile contact depth maps directly from visual input and fusing these generated tactile cues with vision through contrastive learning. The framework comprises VT-Gen, an encoder-decoder that produces contact depth images from visual sequences, and VT-Con, an RL module that integrates visual and generated tactile features via MoCo-based contrastive learning and attention-based fusion. Trained in simulation with a two-stage pipeline, ViTacGen enables zero-shot deployment on visual-only robots and achieves up to 86% success in real-world pushing tasks, outperforming vision-only and tactile baselines. The work offers a practical path toward tactile-aware manipulation without expensive sensors, though it uses contact depth as a high-level tactile proxy and suggests future work to model richer tactile properties and shear forces from vision.

Abstract

Robotic pushing is a fundamental manipulation task that requires tactile feedback to capture subtle contact forces and dynamics between the end-effector and the object. However, real tactile sensors often face hardware limitations such as high costs and fragility, and deployment challenges involving calibration and variations between different sensors, while vision-only policies struggle with satisfactory performance. Inspired by humans' ability to infer tactile states from vision, we propose ViTacGen, a novel robot manipulation framework designed for visual robotic pushing with vision-to-touch generation in reinforcement learning to eliminate the reliance on high-resolution real tactile sensors, enabling effective zero-shot deployment on visual-only robotic systems. Specifically, ViTacGen consists of an encoder-decoder vision-to-touch generation network that generates contact depth images, a standardized tactile representation, directly from visual image sequence, followed by a reinforcement learning policy that fuses visual-tactile data with contrastive learning based on visual and generated tactile observations. We validate the effectiveness of our approach in both simulation and real world experiments, demonstrating its superior performance and achieving a success rate of up to 86\%.

Figures (5)

• Figure 1: The workflow of our proposed ViTacGen comprises two components: a VT-Gen for vision-to-touch generation, and a VT-Con for reinforcement learning on visual and generated tactile contact depth images with contrastive learning, and is trained in two stages: (a) In simulation, we collect paired visual and tactile data from expert trajectories by a pre-trained RL network lygerakis2024m2curl with visual and tactile observations, and use them to train a vision-to-touch generation network VT-Gen so as to generate contact depth images as a standardized tactile representation from visual input, and (b) we incorporate the frozen VT-Gen to train a goal-conditioned RL policy VT-Con based on proprioception i.e., robot's TCP coordinates, visual, and generated tactile observations. This architecture enables ViTacGen to perform zero-shot deployment in real world using only visual information during inference. Please note that we employ contact depth church2022tactilegym to represent our tactile images following lin2022tactilegym2 (see Sec. \ref{['sec:tactile_gym_simulator']} for details).
• Figure 2: To better fuse the encoded visual and tactile features, we optimize the visual and tactile encoders in VT-Con (Fig. \ref{['fig:pipeline']} (b)) with MoCo he2020moco for contrastive learning between visual and tactile modalities. We utilize two momentum-based encoders to obtain momentum visual and tactile features, which are then aligned with the original features of the other modality through the InfoNCE loss oord2018infonce.
• Figure 3: Real world objects used in our experiments: (a) tea box, (b) meat can, (c) mug, and (d) unseen objects for zero-shot test including i) olive jar, ii) apple, iii) coffee can, iv) soup can, and v) ceramic cup.
• Figure 4: Qualitative results of our ViTacGen in simulation, where we present a tea box and a mug with a curve (sine) trajectories. (i) Visual input with visualized pushing trajectories, where the green points refer to the target trajectory and the red points refer to the successfully executed object trajectory (ii) tactile contact depth prediction generated by VT-Gen (iii) tactile observation. Tactile contact depth maps correspond to frontal end-effector observation.
• Figure 5: Demonstration of ViTacGen in real world scenarios, where we present a tea box and a mug with a curve (sine) trajectories. The trajectories are visualized in simulation and projected onto both simulated and real world environments to demonstrate the system's performance. (i) Simulation visual sequences for reference (ii) real world visual sequences (iii) tactile contact depth images generated from real world visual input, corresponding to frontal end-effector observation. More qualitative results are available in our supplementary material.