Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Continuous Control with Geometric Regularity from Robot Intrinsic Symmetry

Shengchao Yan, Baohe Zhang, Yuan Zhang, Joschka Boedecker, Wolfram Burgard

TL;DR

The paper addresses the challenge of learning control policies for high-DOF robots by exploiting intrinsic geometric symmetry. It reformulates single-robot control as a cooperative MARL problem, partitioning the robot into symmetric parts and enforcing geometric regularity through equivariant policy networks and invariant critics with parameter sharing. The MASA architecture demonstrates strong, data-efficient performance across online PPO-based and offline BC/IQL-based settings on five challenging tasks in simulation and real robots, with larger gains as task difficulty grows. This approach offers a scalable pathway to leverage robot symmetry for more sample-efficient and robust learning in complex robotic systems.

Abstract

Geometric regularity, which leverages data symmetry, has been successfully incorporated into deep learning architectures such as CNNs, RNNs, GNNs, and Transformers. While this concept has been widely applied in robotics to address the curse of dimensionality when learning from high-dimensional data, the inherent reflectional and rotational symmetry of robot structures has not been adequately explored. Drawing inspiration from cooperative multi-agent reinforcement learning, we introduce novel network structures for single-agent control learning that explicitly capture these symmetries. Moreover, we investigate the relationship between the geometric prior and the concept of Parameter Sharing in multi-agent reinforcement learning. Last but not the least, we implement the proposed framework in online and offline learning methods to demonstrate its ease of use. Through experiments conducted on various challenging continuous control tasks on simulators and real robots, we highlight the significant potential of the proposed geometric regularity in enhancing robot learning capabilities.

Learning Continuous Control with Geometric Regularity from Robot Intrinsic Symmetry

TL;DR

The paper addresses the challenge of learning control policies for high-DOF robots by exploiting intrinsic geometric symmetry. It reformulates single-robot control as a cooperative MARL problem, partitioning the robot into symmetric parts and enforcing geometric regularity through equivariant policy networks and invariant critics with parameter sharing. The MASA architecture demonstrates strong, data-efficient performance across online PPO-based and offline BC/IQL-based settings on five challenging tasks in simulation and real robots, with larger gains as task difficulty grows. This approach offers a scalable pathway to leverage robot symmetry for more sample-efficient and robust learning in complex robotic systems.

Abstract

Geometric regularity, which leverages data symmetry, has been successfully incorporated into deep learning architectures such as CNNs, RNNs, GNNs, and Transformers. While this concept has been widely applied in robotics to address the curse of dimensionality when learning from high-dimensional data, the inherent reflectional and rotational symmetry of robot structures has not been adequately explored. Drawing inspiration from cooperative multi-agent reinforcement learning, we introduce novel network structures for single-agent control learning that explicitly capture these symmetries. Moreover, we investigate the relationship between the geometric prior and the concept of Parameter Sharing in multi-agent reinforcement learning. Last but not the least, we implement the proposed framework in online and offline learning methods to demonstrate its ease of use. Through experiments conducted on various challenging continuous control tasks on simulators and real robots, we highlight the significant potential of the proposed geometric regularity in enhancing robot learning capabilities.
Paper Structure (25 sections, 4 equations, 4 figures, 3 tables)

This paper contains 25 sections, 4 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Multi-Agent Reinforcement Learning (MARL)
  4. Symmetry in Robot Learning
  5. Single robot Control as MARL
  6. Dividing Single Robots into Multiple Parts
  7. Multi-Agent Reinforcement Learning Formulation
  8. Geometric Regularization
  9. Experiments
  10. Experiments with Online Reinforcement Learning
  11. Baselines
  12. Results
  13. Experiments with Offline Reinforcement Learning
  14. System Setup
  15. Baselines
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Tasks challenging for current deep reinforcement learning baseline algorithms.
  • Figure 2: Agent partitioning considering symmetry structures: Humanoid and Cheetah robots split into left and right parts by reflectional symmetry; TriFinger and Ant robots split into three and four parts by rotational symmetry, where each part is controlled individually by a dedicated agent. The central part (grey) is controlled by all agents.
  • Figure 3: a) TriFinger robot moves an object towards a target position. The black coordinate system is the global system, while the colored ones are local systems. The red arrow represents the desired moving direction of the manipulated object. Note that the actions of different body parts should be equivariant with regard to the rotations. b) Equivariant policy network with parameter $\Phi$. c and s stand for central and symmetric actions. c) Invariant critic network with parameter $\Psi,\Theta$.
  • Figure 4: Learning curves on robot control tasks. The x-axis is environment time steps and the y-axis is episodic returns during training. All graphs are plotted with median and 25%-75% percentile shading across five random seeds.

Theorems & Definitions (1)

  • proof : Proof of the network equivariance/invariance