Probabilistic Surface Friction Estimation Based on Visual and Haptic Measurements

TL;DR

This work addresses the challenge of estimating local surface friction on objects that comprise multiple materials by fusing visual cues with limited haptic exploration. It introduces a probabilistic visuo-haptic model built as a $C$-component Gaussian Mixture Model over the joint features $\mathcal{P}(\mathbf{V},\mathbf{H})$, with $C=n+1$ (one per material plus a background) and uses Gaussian Mixture Regression to infer the conditional $\mathcal{P}(\mathbf{H}|\mathbf{V})$ for unseen regions, yielding both mean friction estimates and uncertainty. The approach is implemented on a real robot, enabling friction maps to be generated for 15 objects (single- and multi-material) and used to guide grasp planning toward high-friction areas, thereby improving grasp success. By explicitly modeling nonuniform friction and providing uncertainty-aware predictions, the method advances tactile-visual integration for robust manipulation in realistic environments.

Abstract

Accurately modeling local surface properties of objects is crucial to many robotic applications, from grasping to material recognition. Surface properties like friction are however difficult to estimate, as visual observation of the object does not convey enough information over these properties. In contrast, haptic exploration is time consuming as it only provides information relevant to the explored parts of the object. In this work, we propose a joint visuo-haptic object model that enables the estimation of surface friction coefficient over an entire object by exploiting the correlation of visual and haptic information, together with a limited haptic exploration by a robotic arm. We demonstrate the validity of the proposed method by showing its ability to estimate varying friction coefficients on a range of real multi-material objects. Furthermore, we illustrate how the estimated friction coefficients can improve grasping success rate by guiding a grasp planner toward high friction areas.

Figures (8)

• Figure 1: After the shape of an object has been captured from a camera, surface friction on a small part of the object is estimated using haptic exploration. The friction over the entire object is then predicted by coupling visual information with the local haptic measurements.
• Figure 2: The proposed pipeline: the object's visual properties are first acquired as point-cloud, which is then filtered and pre-segmented into regions. The robot then performs an haptic exploration over some of the regions. The proposed model then is used to estimate the friction coefficient over the whole object, together with the corresponding confidence.
• Figure 3: A qualitative comparison between friction models. The orange line denotes the friction coefficient estimated using Coloumb friction model, while the blue line denotes the one using LuGre friction model (best viewed in color).
• Figure 4: The experimental setup
• Figure 5: Real objects, along with estimated friction coefficient and uncertainty returned by the proposed method. Red indicates higher friction coefficient value, while green denotes higher uncertainty. The haptic exploration is showed by the path in green (best viewed in color).