NeuralTouch: Neural Descriptors for Precise Sim-to-Real Tactile Robot Control

TL;DR

NeuralTouch presents a two-stage grasping paradigm that combines Neural Descriptor Fields (NDF) with a tactile reinforcement learning policy to achieve precise, generalizable manipulation. The coarse-grain pose is proposed by NDF, while a tactile RL policy conditioned on neural pose descriptors refines the grasp through in-hand touch, enabling robust performance across unseen objects and contact geometries. Key contributions include a descriptor-conditioned tactile policy, a unified treatment of semantic grasp poses, and successful zero-shot sim-to-real transfer demonstrated on bottle lid opening and peg-in-hole tasks. This work advances practical, contact-rich robotic manipulation by leveraging multimodal sensing and a generalizable descriptor space to bridge vision and touch in real-world settings.

Abstract

Grasping accuracy is a critical prerequisite for precise object manipulation, often requiring careful alignment between the robot hand and object. Neural Descriptor Fields (NDF) offer a promising vision-based method to generate grasping poses that generalize across object categories. However, NDF alone can produce inaccurate poses due to imperfect camera calibration, incomplete point clouds, and object variability. Meanwhile, tactile sensing enables more precise contact, but existing approaches typically learn policies limited to simple, predefined contact geometries. In this work, we introduce NeuralTouch, a multimodal framework that integrates NDF and tactile sensing to enable accurate, generalizable grasping through gentle physical interaction. Our approach leverages NDF to implicitly represent the target contact geometry, from which a deep reinforcement learning (RL) policy is trained to refine the grasp using tactile feedback. This policy is conditioned on the neural descriptors and does not require explicit specification of contact types. We validate NeuralTouch through ablation studies in simulation and zero-shot transfer to real-world manipulation tasks--such as peg-out-in-hole and bottle lid opening--without additional fine-tuning. Results show that NeuralTouch significantly improves grasping accuracy and robustness over baseline methods, offering a general framework for precise, contact-rich robotic manipulation.

Figures (8)

• Figure 1: With just a few demonstrations of a manipulation task in simulation, our method NeuralTouch effectively generalizes to achieve accurate grasping poses for unseen objects in real-world settings, utilizing both vision and touch.
• Figure 2: Overview of the NeuralTouch: In simulation, we first pre-train an occupancy network which is the core component of the Neural Pose Descriptor Fields. Secondly, we collect human demonstrations along with object point clouds and robot target grasping pose descriptors depending on the manipulation tasks. Thirdly, we train an RL policy with tactile and proprioceptive feedback, to achieve fine grasping poses implicitly specified by these collected descriptors. After obtaining the NPDF and a well-trained policy, our system is directly deployed in the real world with a real-to-sim tactile transfer to accurately grasp unseen objects, executing manipulation tasks such as unplugging a bolt-like USB and inserting it into a socket.
• Figure 3: We use three different types of objects for occupancy network training: bolts, mugs, and bottles.
• Figure 4: The objects used in the two proposed manipulation tasks for real-world testing (from left to right): an apple juice bottle, a syrup container, a bottle of ketchup, a bolt, a plug adaptor, a USB.
• Figure 5: Comparison of training performances for NeuralTouch (blue) and vanilla RL-Touch (red) across 10 random seeds. (a) NeuralTouch, leveraging neural descriptors from NDF, achieves superior asymptotic performance (higher average episode rewards), whereas vanilla RL-Touch struggles. (b) NeuralTouch completes tasks with significantly shorter episode lengths, demonstrating enhanced efficiency compared to RL policies without NDF.