Action Conditioned Tactile Prediction: case study on slip prediction

TL;DR

This work addresses tactile prediction during real-world robot manipulation by proposing two action-conditioned models, ACTP and ACTVP, trained on a large tactile dataset collected with the Xela uSkin magnetic sensor. The authors compare these models against state-of-the-art video-based prediction methods and baselines, evaluating on quantitative metrics (MAE, PSNR, SSIM), qualitative fidelity, and slip prediction performance. Key findings show ACTVP delivers the best general predictive accuracy, while ACTP excels in slip anticipation, and that optical-flow and encoder-decoder approaches do not consistently improve tactile prediction on this dataset. The study provides a valuable dataset and benchmarking framework, highlighting the potential of action-conditioned tactile prediction to support slip-avoidance and other manipulation tasks, while also outlining limitations and avenues for future work with higher-resolution or multi-modal sensing.

Abstract

Tactile predictive models can be useful across several robotic manipulation tasks, e.g. robotic pushing, robotic grasping, slip avoidance, and in-hand manipulation. However, available tactile prediction models are mostly studied for image-based tactile sensors and there is no comparison study indicating the best performing models. In this paper, we presented two novel data-driven action-conditioned models for predicting tactile signals during real-world physical robot interaction tasks (1) action condition tactile prediction and (2) action conditioned tactile-video prediction models. We use a magnetic-based tactile sensor that is challenging to analyse and test state-of-the-art predictive models and the only existing bespoke tactile prediction model. We compare the performance of these models with those of our proposed models. We perform the comparison study using our novel tactile-enabled dataset containing 51,000 tactile frames of a real-world robotic manipulation task with 11 flat-surfaced household objects. Our experimental results demonstrate the superiority of our proposed tactile prediction models in terms of qualitative, quantitative and slip prediction scores.

Figures (10)

• Figure 1: (A) Teleoperated kinesthetic data collection with tactile finger tipped robot (B) Xela uSkin tactile sensor (C) Single taxel value during pick and move trial and ACTP tactile signal prediction. Letters and vertical bars indicating correct peak and trough predictions ahead of time.
• Figure 2: (A) Teleoperated data collection set-up. The left robot is the 'follower', grasping the object with tactile sensing fingers. The right robot is the 'leader' and is teleoperated by human control (B) Eleven household box shaped objects used for training and testing, including tissue box, toothpaste box, and chopped tomatoes. Object set has variance in size, weight, centre of mass, material, stiffness and contact properties. Markers can be seen on the top and side of the objects, these are used to localise the object between the robot fingers, which is used for slip classification (C) Dataset example of a full trial.
• Figure 3: Tactile prediction model architectures (left) Action Conditioned Tactile Prediction (ACTP) and (right) Action Conditioned Tactile-Video Prediction (ACTVP)
• Figure 4: Tactile predictions at prediction time-step (top) examples the perfect tactile prediction model for reference with Figures \ref{['fig::intro_image']}, \ref{['fig::t10plotsImages']}, \ref{['fig::ACcomparison']}, \ref{['fig::NAvsA']}, \ref{['fig::ACTPvsACTVP']} and \ref{['fig::classifier']}. (bottom) CDNA predictions, showing poor performance, especially at extended time horizons.
• Figure 5: Comparison between SVG and ACTVP, showing the poor performance of SVG's t+10 predictions, this level of performance is also indicated by the performance metric results shown in Table \ref{['tab:full_perf_table']}.