On-Robot Learning With Equivariant Models

TL;DR

The paper demonstrates that on-robot policy learning can be dramatically accelerated by using equivariant neural networks with discrete symmetry groups and simple data augmentation, enabling several non-trivial manipulation tasks to be learned directly on hardware in a few hours. It shows that discrete groups like D4/D8 outperform continuous symmetry variants, and that buffer-based rotation augmentation further boosts performance. Compared to sim2real approaches and a strong baseline (FERM), on-robot training with Equivariant SAC often achieves higher final performance and can avoid negative transfer associated with pretraining in simulation. The work underscores the practical potential of symmetry-aware RL for rapid, hardware-centric skill acquisition while outlining limitations and directions for automating symmetry discovery and expanding modalities.

Abstract

Recently, equivariant neural network models have been shown to improve sample efficiency for tasks in computer vision and reinforcement learning. This paper explores this idea in the context of on-robot policy learning in which a policy must be learned entirely on a physical robotic system without reference to a model, a simulator, or an offline dataset. We focus on applications of Equivariant SAC to robotic manipulation and explore a number of variations of the algorithm. Ultimately, we demonstrate the ability to learn several non-trivial manipulation tasks completely through on-robot experiences in less than an hour or two of wall clock time.

Figures (16)

• Figure 1: Illustration of the Equivariant SAC. (a): the equivariant actor's output action rotates as the input state rotates. (b): the invariant critic's output doesn't change when the input state and action are rotated simultaneously.
• Figure 2: (a)-(d): Our simulation environments implemented in PyBullet pybullet. The left images in each environment show the initial state of the environment; the right images in each environment show the goal state. (e)-(h): Our on-robot learning environments.
• Figure 3: The plots show the performance of the behavior policy in terms of the discounted reward. Each point is the average discounted reward in the previous 500 steps. Results are averaged over four runs. Shading denotes standard error.
• Figure 4: Our experimental set up for on-robot learning. The observation (bottom right) is generated by first acquiring point clouds from two depth cameras above the workspace then creating an orthographic projection at the gripper's position. The gripper is drawn at the center of the observation (in yellow) with its current aperture and orientation.
• Figure 5: Comparison of Equivariant SAC trained from scratch (blue), Equivariant SAC with sim-to-real fine-tuning (green), and FERM (red) in real world. The plots show the performance of the behavior policy in terms of the discounted reward. Each point is the average discounted reward in the previous 200 steps. Results are averaged over three runs. Shading denotes standard error.