Enhancing Tactile-based Reinforcement Learning for Robotic Control

TL;DR

This work addresses the challenge of making tactile sensing reliable in reinforcement learning by proposing four self-supervised objectives—Tactile Reconstruction, Full Reconstruction, Forward Dynamics, and Tactile Forward Dynamics—applied to a proprioception plus sparse binary tactile setup. Modeling tactile manipulation as a POMDP, the approach learns a compact latent representation $\mathbf{z_t}$ via an observation encoder, which powers a PPO policy and value function while optimizing SSL losses. Empirical results on the RoTO benchmark (Find, Bounce, Baoding) show that SSL-based tactile learning substantially improves performance over RL-only baselines, with forward dynamics providing strong, sometimes task-dependent gains, and a separated auxiliary memory stabilizing training for long-horizon tasks like Baoding. The work introduces a scalable, vision-free tactile RL paradigm that achieves superhuman dexterity in simulation and offers practical guidance for deploying tactile-focused SSL in real-world robotic manipulation, while acknowledging limitations such as lack of real hardware validation and increased compute time.

Abstract

Achieving safe, reliable real-world robotic manipulation requires agents to evolve beyond vision and incorporate tactile sensing to overcome sensory deficits and reliance on idealised state information. Despite its potential, the efficacy of tactile sensing in reinforcement learning (RL) remains inconsistent. We address this by developing self-supervised learning (SSL) methodologies to more effectively harness tactile observations, focusing on a scalable setup of proprioception and sparse binary contacts. We empirically demonstrate that sparse binary tactile signals are critical for dexterity, particularly for interactions that proprioceptive control errors do not register, such as decoupled robot-object motions. Our agents achieve superhuman dexterity in complex contact tasks (ball bouncing and Baoding ball rotation). Furthermore, we find that decoupling the SSL memory from the on-policy memory can improve performance. We release the Robot Tactile Olympiad (RoTO) benchmark to standardise and promote future research in tactile-based manipulation. Project page: https://elle-miller.github.io/tactile_rl

Figures (20)

• Figure 1: Tactile-based RL with self-supervision. Our agents achieve superhuman dexterity in our RoTO benchmark using only a history of proprioception and binary contacts (i.e., with no vision or privileged information). By jointly training the observation encoder with self-supervision, the agent transforms complex sensory input into representations that capture object positions and velocities.
• Figure 2: Proposed self-supervised objectives. Reconstruction losses (TR, FR) force the encoder to preserve tactile information within $\mathbf{z_t}$ by using a binary classification decoder to reconstruct the original $\mathbf{o_t^{tact}}$. Forward dynamics losses (FD, TFD) ensures the encoder extracts the necessary information from $\mathbf{o_t}$ for accurate forecasting of the representation $\mathbf{z_t}$ several timesteps into the future.
• Figure 3: RL-only. Mean evaluation returns of proprioceptive-tactile ($\mathbf{o^{prop}} \oplus \mathbf{o^{tact}}$) and proprioceptive ($\mathbf{o^{prop}}$) agents. We run one seed with the last action $\mathbf{a_{t-1}}$ removed from the $\mathbf{o^{prop}}$ agent.
• Figure 4: RL+SSL. Mean evaluation returns of the self-supervised agents.
• Figure 5: RL+SSL+Memory. Mean evaluation returns of the FD agent trained on the on-policy rollout vs $N_{rollouts}$ last rollouts. The effect is minimal in Find and Bounce but substantial in Baoding.