Deep Domain Adaptation Regression for Force Calibration of Optical Tactile Sensors

TL;DR

This work tackles the problem of calibrating 3D contact forces for optical tactile sensors across aging and variant sensors without extensive labeled data. It introduces a deep domain adaptation regression framework that learns a shared regressor while aligning feature spaces between labeled source and unlabeled target domains via Local Maximum Mean Discrepancy (LMMD) and a pseudo-label–assisted classification head. By explicitly studying three domain gaps—marker presence, illumination, and elastomer modulus—the approach demonstrates substantial reductions in force-prediction errors (e.g., normal force MAE ~0.102 N, ~3.4% of full range) and meaningful improvements for shear forces, outperforming baseline source-only transfers and other DA methods. The method eliminates the need for force/torque measurement tools during calibration and speeds up deployment to new or aging sensors, with results validated on GelSight-type tactile sensors and broad applicability to similar optical tactile systems.

Abstract

Optical tactile sensors provide robots with rich force information for robot grasping in unstructured environments. The fast and accurate calibration of three-dimensional contact forces holds significance for new sensors and existing tactile sensors which may have incurred damage or aging. However, the conventional neural-network-based force calibration method necessitates a large volume of force-labeled tactile images to minimize force prediction errors, with the need for accurate Force/Torque measurement tools as well as a time-consuming data collection process. To address this challenge, we propose a novel deep domain-adaptation force calibration method, designed to transfer the force prediction ability from a calibrated optical tactile sensor to uncalibrated ones with various combinations of domain gaps, including marker presence, illumination condition, and elastomer modulus. Experimental results show the effectiveness of the proposed unsupervised force calibration method, with lowest force prediction errors of 0.102N (3.4\% in full force range) for normal force, and 0.095N (6.3\%) and 0.062N (4.1\%) for shear forces along the x-axis and y-axis, respectively. This study presents a promising, general force calibration methodology for optical tactile sensors.

Figures (5)

• Figure 1: Deep domain adaptation for force calibration of optical tactile sensors. Upon completing domain adaptation, the feature space of the unlabeled tactile images $I_t$ from the target domain and the labeled tactile images $I_s$ from the source domain are aligned. A shared regressor trained with ground truth forces $F^s$ in source domain can be used to predict $F^t$ in target domain. $I_t^1$, $I_t^2$ and $I_t^3$ denote tactile images in target domain with varying combinations of domain gaps. $F_T$ represents the total force, $F_z$ denotes the normal force, $F_x$ and $F_y$ denote the shear forces, and $\theta$ represents the force angle, respectively.
• Figure 2: Deep domain adaptation regression model for force calibration. The model comprises four components: a feature encoder, a contact classification head, a force regression head, and a domain transfer head. The model takes batches of tactile images from the source domain $\mathbf{I}_s$ and the target domain $\mathbf{I}_t$ as inputs. Both $\mathbf{I}_s$ and $\mathbf{I}_t$ consist of concatenated tactile images comprising a current contact image $\textup{I}^c$ and a reference image $\textup{I}^r$. The domain transfer head accepts features ($\mathbf{f}_s$ and $\mathbf{f}_t$) and ground truth contact labels $\mathbf{C}_s$ from the source domain, as well as the pseudo labels $\mathbf{C}_t$ predicted from the classifier. The overall loss $\mathcal{L}_h$ is weighted sum of the regression loss $\mathcal{L}_r$, classification loss $\mathcal{L}_c$ and domain adaptation loss $\mathcal{L}_t$.
• Figure 3: (a) Data collection setup with a robot arm, a GelSight sensor, a sphere indenter and a Nano17 F/T sensor. (b) Programmed contact path for data collection. Contact pixel denotes a point for indentation. (c) Real-world setup. (d) Four groups of collected tactile images with different combinations of domain gaps.
• Figure 4: Force prediction errors compared among domain adaptation groups featuring one variable (a-b), two variables (c-d), and three variables (e-f). This comparison is conducted using both the source-only method (i) and our domain adaptation method (ii).
• Figure 5: Feature spaces visualized using tSNE from the source-only method and our domain adaptation method in (a) one domain variable, (b) two domain variables, (c) and three domain variables.