VT-Refine: Learning Bimanual Assembly with Visuo-Tactile Feedback via Simulation Fine-Tuning

TL;DR

VT-Refine addresses the challenge of precise bimanual assembly by combining real-world visuo-tactile demonstrations with large-scale simulation-based fine-tuning and real-to-real transfer. It pre-trains a diffusion policy on synchronized visual and tactile data, then refines it in a GPU-accelerated tactile simulation using DPPO in a digital twin before deploying back to the real world. The approach employs a unified visuo-tactile point-cloud representation and a tactile calibration pipeline to enable robust sim-to-real transfer, achieving substantial performance gains across five tasks and two robot platforms. The method demonstrates that RL fine-tuning leverages tactile feedback to produce micro-adjustments and robust insertions, reducing the sim-to-real gap and offering a scalable path for contact-rich manipulation. Overall, VT-Refine provides a data-efficient, transferable framework for two-handed manipulation with tactile sensing that can generalize across tasks and domains.

Abstract

Humans excel at bimanual assembly tasks by adapting to rich tactile feedback -- a capability that remains difficult to replicate in robots through behavioral cloning alone, due to the suboptimality and limited diversity of human demonstrations. In this work, we present VT-Refine, a visuo-tactile policy learning framework that combines real-world demonstrations, high-fidelity tactile simulation, and reinforcement learning to tackle precise, contact-rich bimanual assembly. We begin by training a diffusion policy on a small set of demonstrations using synchronized visual and tactile inputs. This policy is then transferred to a simulated digital twin equipped with simulated tactile sensors and further refined via large-scale reinforcement learning to enhance robustness and generalization. To enable accurate sim-to-real transfer, we leverage high-resolution piezoresistive tactile sensors that provide normal force signals and can be realistically modeled in parallel using GPU-accelerated simulation. Experimental results show that VT-Refine improves assembly performance in both simulation and the real world by increasing data diversity and enabling more effective policy fine-tuning. Our project page is available at https://binghao-huang.github.io/vt_refine/.

Figures (10)

• Figure 1: We propose VT-Refine, a novel visuo-tactile policy learning framework for precise, contact-rich bimanual assembly tasks. Top Left: We collect real-world demonstrations and pre-train a diffusion policy using visuo-tactile inputs. Right to Bottom Left: We leverage tactile simulation and large-scale reinforcement learning to fine-tune the policy, and subsequently transfer the fine-tuned policy back to the real world. The resulting policy demonstrates strong performance in both simulated and real environments.
• Figure 2: Tactile Sensing in Real and Simulation. (a) Our real-world hardware setup, including the design of the piezoresistive tactile sensor. Four tactile sensor pads (two per hand) are mounted on the soft gripper to capture contact forces. (b) Replication of the tactile sensing process in simulation. A spring-damper model is used to simulate the interaction between the tactile points and objects to generate realistic tactile signals.
• Figure 3: Two-Stage Visuo-Tactile Policy Training.Stage 1: We collect real-world human demonstrations with visual and tactile modalities and pre-train a diffusion policy. Stage 2: We simulate the same sensory modalities in simulation and fine-tune the pre-trained diffusion policy using policy-gradient-based RL.
• Figure 4: Sensor Calibration Results. The histogram shows consistent sensor reading distributions between the simulation and real world.
• Figure 5: Real Robot Setups. Both the socket and plug are randomly placed in a $3cm \times 3cm$ area. Both robot setups have four tactile sensing pads and an ego-centric camera.