Touch2Shape: Touch-Conditioned 3D Diffusion for Shape Exploration and Reconstruction

TL;DR

Touch2Shape introduces a touch-conditioned diffusion framework for active 3D shape exploration and reconstruction from tactile sensing. It leverages a low-dimensional latent $z$ from a pretrained VQ-VAE, a contrastive touch encoder, and a touch-shape fusion module to refine local geometry, while an RL-based policy guides touch locations using diffusion-loss-based rewards. The approach achieves superior reconstruction quality under tactile-only and visual-tactile settings and demonstrates the value of active touch for revealing fine details. The work lays groundwork for real-world robotic object exploration and extension to neural rendering for multi-view synthesis.

Abstract

Diffusion models have made breakthroughs in 3D generation tasks. Current 3D diffusion models focus on reconstructing target shape from images or a set of partial observations. While excelling in global context understanding, they struggle to capture the local details of complex shapes and limited to the occlusion and lighting conditions. To overcome these limitations, we utilize tactile images to capture the local 3D information and propose a Touch2Shape model, which leverages a touch-conditioned diffusion model to explore and reconstruct the target shape from touch. For shape reconstruction, we have developed a touch embedding module to condition the diffusion model in creating a compact representation and a touch shape fusion module to refine the reconstructed shape. For shape exploration, we combine the diffusion model with reinforcement learning to train a policy. This involves using the generated latent vector from the diffusion model to guide the touch exploration policy training through a novel reward design. Experiments validate the reconstruction quality thorough both qualitatively and quantitative analysis, and our touch exploration policy further boosts reconstruction performance.

Figures (6)

• Figure 1: (1) Exploring the target object and capturing the tactile image to reconstruct the 3D shape. We trained a diffusion model to obtain a low-dimensional and compact latent vector, which is used for predicting next touch location and reconstructing the target shapes. The numbers ① with black arrows and ② with greeen arrows in the diagram represent two consecutive time steps. We only generate the full reconstruction at the final step. (2) As the touch exploration progresses, the reconstruction results gradually approach the ground truth. (3) Reconstruction results compared with ActiveVT activevt (under visual-tactile settings).
• Figure 2: We pretrained (a) the shape encoder and decoder, (b) the touch CNN model that is used for touch chart prediction, and (c) the contrastive touch encoder. During the training of (d) the touch-conditioned 3D diffusion model, we kept the parameters of the pretrained modules fixed and train a touch-conditioned 3D diffusion model for touch exploration policy learning and shape reconstruction.
• Figure 3: Touch shape fusion module. The black arrows indicate the flow of the shape decoder, while the red arrows represent the flow after incorporating the touch shape fusion module. To simplify, we use 2D grids to visualize the 3D feature maps.
• Figure 4: Qualitative results of ActiveVT activevt and ours. While ActiveVT struggles with visualizations and detail preservation, our method excels in maintaining global shape accuracy across diverse structures, ensuring satisfactory local details.
• Figure 5: The evolution of the reconstructed shape with an increasing number of grasps (in the tactile only setting). Initially, limited information makes determining the overall global shape challenging, but with more grasp actions, our method effectively improves the reconstruction quality. The local points sampled on predicted touch charts (from captrued valid tactile images) are painted blue.