Tactile-based force estimation for interaction control with robot fingers

TL;DR

This work tackles real-time force estimation from full tactile sensor arrays mounted on a robotic hand to enable reactive interaction control. It introduces a data-efficient on-hand calibration framework that reconstructs 3D contact forces across curved and flat sensor geometries by evaluating five force-estimation models (M1–M5) and selecting robust candidates (M3λ and M4) for online use. In online and closed-loop experiments, the M3λ model demonstrates strong generalization to unseen objects and surfaces, while M4 excels on distributions similar to training data, achieving a ~74% reduction in true tracking error versus open-loop and maintaining low force-tracking errors around 0.12–0.17 N. The approach removes the need for sensor pre-calibration, supports high-bandwidth (100 Hz) force control, and holds promise for scalable tactile-based manipulation across diverse tasks including soft-object handling and teleoperation.

Abstract

Fine dexterous manipulation requires reactive control based on rich sensing of manipulator-object interactions. Tactile sensing arrays provide rich contact information across the manipulator's surface. However their implementation faces two main challenges: accurate force estimation across complex surfaces like robotic hands, and integration of these estimates into reactive control loops. We present a data-efficient calibration method that enables rapid, full-array force estimation across varying geometries, providing online feedback that accounts for non-linearities and deformation effects. Our force estimation model serves as feedback in an online closed-loop control system for interaction force tracking. The accuracy of our estimates is independently validated against measurements from a calibrated force-torque sensor. Using the Allegro Hand equipped with Xela uSkin sensors, we demonstrate precise force application through an admittance control loop running at 100Hz, achieving up to 0.12+/-0.08 [N] error margin-results that show promising potential for dexterous manipulation.

Figures (5)

• Figure 1: The tactile observation $\boldsymbol{x}$ is mapped by function $\boldsymbol{\mathcal{H}}$ to estimate interaction force $\boldsymbol{\hat{f}}$. This $\boldsymbol{\hat{f}}$ is fed back to the force controller, to adjust the force's orientation and magnitude, ensuring the finger applies desired force $\boldsymbol{f}_{d}$. The measured force $\boldsymbol{\tilde{f}}$ serves as ground truth for the estimator.
• Figure 2: Fingertip with a curved tactile array. Each taxel (30 here) measures its activation in local frame $\{S^{(i)}\}$; $\{S^{(0)}\}$ is the finger tip frame. Each RGB frame represents a taxel.
• Figure 3: (a) Hardware setup: we use a uSkin covered Allegro hand and an ATI 45mini force torque sensor for calibration. Both are mounted on a rigid chassis. (b) Interaction control setup: Object inserted between fingertip and reference force torque sensor. (c) Selected YCB objects: Foam brick, Softball, Abrasive sponge and Bleach cleanser. (d) Push-plates: Button, Flat, Fillet, Convex, Edge and Spherical.
• Figure 4: Closed loop task space force controller. The tactile-equipped hand is in rigid contact with the environment; $\boldsymbol{x}$ is the tactile observation and $\boldsymbol{\mathcal{H}}$ the function s. that $\boldsymbol{\mathcal{H}}(\boldsymbol{x}) = \boldsymbol{\hat{f}}$.
• Figure 5: (Left) Closed loop force controller performances. The force estimation is computed with $M3\lambda$. (Right) Closed loop force controller performances, with a soft object in between the finger and the force sensor (YCB foam brick). The force estimation is computed with $M3\lambda$. Each experiment is repeated 5 times, the solid line represents the average and the aura represents the standard deviation