CLTP: Contrastive Language-Tactile Pre-training for 3D Contact Geometry Understanding

TL;DR

CLTP introduces a contrastive language–tactile pretraining framework that aligns tactile 3D point clouds with natural language to capture rich contact states for robotic manipulation. Leveraging the TCL3D dataset (52k+ samples across 117 objects) and a frozen vision–language space, CLTP uses two contrastive losses to align tactile, textual, and visual modalities, enabling zero-shot 3D touch classification, standard contact state classification, and Tac3D-LLM interactions. The approach yields strong sim-to-real transfer, outperforms vision-based tactile baselines, and enables LLM-driven tactile reasoning for tasks such as precise strawberry positioning. By open-sourcing code and data, the work provides a scalable path toward tactile-language-action models that understand contact geometry and force in manipulation tasks.

Abstract

Recent advancements in integrating tactile sensing with vision-language models (VLMs) have demonstrated remarkable potential for robotic multimodal perception. However, existing tactile descriptions remain limited to superficial attributes like texture, neglecting critical contact states essential for robotic manipulation. To bridge this gap, we propose CLTP, an intuitive and effective language tactile pretraining framework that aligns tactile 3D point clouds with natural language in various contact scenarios, thus enabling contact-state-aware tactile language understanding for contact-rich manipulation tasks. We first collect a novel dataset of 50k+ tactile 3D point cloud-language pairs, where descriptions explicitly capture multidimensional contact states (e.g., contact location, shape, and force) from the tactile sensor's perspective. CLTP leverages a pre-aligned and frozen vision-language feature space to bridge holistic textual and tactile modalities. Experiments validate its superiority in three downstream tasks: zero-shot 3D classification, contact state classification, and tactile 3D large language model (LLM) interaction. To the best of our knowledge, this is the first study to align tactile and language representations from the contact state perspective for manipulation tasks, providing great potential for tactile-language-action model learning. Code and datasets are open-sourced at https://sites.google.com/view/cltp/.

Figures (8)

• Figure 1: Overview of the CLTP framework and its downstream tasks. The left and middle part are the TCL3D dataset construction process and the CLTP pre-training framework. CLTP uses a large multimodal model to automatically generate a detailed text description for each 2D rendered tactile image from the 3D contact deformed tactile point cloud (including contact shape, area, position, depth and texture). CLTP uses a pre-aligned and frozen vision-language feature space to achieve alignment between the three modalities (text, image and 3D tactile point cloud). After pre-training, the 3D encoder will be used for three downstream tasks.
• Figure 2: TCL3D dataset: a multimodal multi-sensor tactile alignment dataset for CLTP training and evaluation.
• Figure 3: Architecture of CLTP framework. It leverages a frozen pre-aligned vision-language feature space to establish connections between tactile and language modalities.
• Figure 4: Case Study: CLTP is more sensitive to the shape, pressing depth, and contact position of the contact 3D point cloud than the baseline models.
• Figure 5: A case study of generating text descriptions from realistic tactile point clouds.