UniForce: A Unified Latent Force Model for Robot Manipulation with Diverse Tactile Sensors

TL;DR

UniForce addresses the hurdle of generalizing force-aware manipulation across diverse tactile sensors by learning a unified latent force space grounded in cross-sensor force equilibrium. It introduces a CVAE-based encoder that maps tactile observations to a patch-wise latent force map and a decoder that performs physics-conditioned forward dynamics to reconstruct tactile observations, enabling zero-shot transfer without sensor-specific force labels. The approach is validated across GelSight, TacTip, and uSkin through latent-space analysis, cross-sensor reconstruction, zero-shot force estimation, and VTLA-based wiping tasks, showing improved cross-sensor force prediction and policy transfer. This work reduces sensor-specific retraining while enabling robust, force-aware manipulation in heterogeneous tactile setups, with practical implications for scalable tactile-integrated robotics.

Abstract

Force sensing is essential for dexterous robot manipulation, but scaling force-aware policy learning is hindered by the heterogeneity of tactile sensors. Differences in sensing principles (e.g., optical vs. magnetic), form factors, and materials typically require sensor-specific data collection, calibration, and model training, thereby limiting generalisability. We propose UniForce, a novel unified tactile representation learning framework that learns a shared latent force space across diverse tactile sensors. UniForce reduces cross-sensor domain shift by jointly modeling inverse dynamics (image-to-force) and forward dynamics (force-to-image), constrained by force equilibrium and image reconstruction losses to produce force-grounded representations. To avoid reliance on expensive external force/torque (F/T) sensors, we exploit static equilibrium and collect force-paired data via direct sensor--object--sensor interactions, enabling cross-sensor alignment with contact force. The resulting universal tactile encoder can be plugged into downstream force-aware robot manipulation tasks with zero-shot transfer, without retraining or finetuning. Extensive experiments on heterogeneous tactile sensors including GelSight, TacTip, and uSkin, demonstrate consistent improvements in force estimation over prior methods, and enable effective cross-sensor coordination in Vision-Tactile-Language-Action (VTLA) models for a robotic wiping task. Code and datasets will be released.

Figures (9)

• Figure 1: We introduce UniForce, a framework that learns a shared latent force model for robot manipulation across heterogeneous tactile sensors. Using paired data from force-equilibrium grasps, UniForce pretrains a universal encoder to extract physically meaningful latent forces from a unified marker-image representation without explicit force labels. The pretrained encoder can then be plugged into downstream tasks by training specific head on data from a single sensor, enabling zero-shot transfer to other sensors without finetuning for cross-sensor force prediction and force-aware policy learning.
• Figure 2: Unified force-aware tactile representation learning. In the inverse-dynamics stage, the encoder takes paired reference/contact observations and infers a patch-wise latent force map; an equilibrium loss aligns left/right latent forces, while a KL term regularizes the posterior. In the forward-dynamics stage, the decoder reconstructs the observation by conditioning on the sensor-specific reference image and the latent force, enabling both self-reconstruction and cross-sensor reconstruction. We mirror right-finger marker images to match the coordinates of the left finger.
• Figure 3: Paired data collection via quasi-static force equilibrium. (i) Data collection setup; (ii) Coordinate alignment by mirroring. (iii--iv) Canonicalizing heterogeneous raw signals referring to chen2026genforce.
• Figure 4: (i) Heatmap of mean Pearson correlation $r$ ($\pm$ SD) between latent dimensions $z_0$–$z_5$ and force components $F_y, F_x, F_z$ (N). Total images $n=294{,}193$ across three sensors; $p<0.001$ (ii) Absolute mean displacement of marker in x-axis and y-axis from UniForce-pair dataset.
• Figure 5: Average zero-shot force prediction errors (MAE). We compare MAE across six heterogeneous sensor-pairs. FT denotes full-training.