A Primer on SO(3) Action Representations in Deep Reinforcement Learning

TL;DR

The paper investigates how SO(3) action representations influence exploration, entropy regularization, and training stability across PPO, SAC, and TD3 in both dense and sparse reward settings. It demonstrates that representing actions as tangent vectors in the local frame (delta tangent actions) yields the most reliable performance across diverse robotics tasks, due to favorable properties of the mapping from network outputs to rotations and better-controlled exploration. The work provides concrete, implementation-ready guidelines for choosing rotation actions and highlights representation-induced effects on learning dynamics and optimization in orientation control. The findings are validated across multiple robot benchmarks, suggesting practical impact for orientation-enabled policies in real-world robotics.

Abstract

Many robotic control tasks require policies to act on orientations, yet the geometry of SO(3) makes this nontrivial. Because SO(3) admits no global, smooth, minimal parameterization, common representations such as Euler angles, quaternions, rotation matrices, and Lie algebra coordinates introduce distinct constraints and failure modes. While these trade-offs are well studied for supervised learning, their implications for actions in reinforcement learning remain unclear. We systematically evaluate SO(3) action representations across three standard continuous control algorithms, PPO, SAC, and TD3, under dense and sparse rewards. We compare how representations shape exploration, interact with entropy regularization, and affect training stability through empirical studies and analyze the implications of different projections for obtaining valid rotations from Euclidean network outputs. Across a suite of robotics benchmarks, we quantify the practical impact of these choices and distill simple, implementation-ready guidelines for selecting and using rotation actions. Our results highlight that representation-induced geometry strongly influences exploration and optimization and show that representing actions as tangent vectors in the local frame yields the most reliable results across algorithms.

Figures (20)

• Figure 1: The agent rotates at max $\alpha_{max}$ radians from the current state ${\bm{R}}_t$ to the next state ${\bm{R}}_{t+1}$ towards the desired state ${\bm{R}}_a$. The goal is to rotate into ${\bm{R}}_{g}$.
• Figure 2: 3D distribution of Euler angles sampled from $\mathcal{N}(0, 2)$ and squashed with $\tanh$ in the Lie algebra $\mathfrak{m}$.
• Figure 3: Achieved reward for the trajectory tracking (left) and drone racing competition (right) tasks across action parameterizations. Shaded areas indicate the standard deviation across 25 seeds.
• Figure 4: Achieved reward across the RoboSuite benchmark as a fraction of the maximum possible reward. Error bars denote the standard deviation across five seeds.
• Figure 5: Achieved reward on ReachOrient (left) and PickAndPlaceOrient (right). Both tangent and matrix action representations converge fast for the reach task, with quaternions second and Euler angles last. On the harder pick and place task, the local tangent space representation significantly outperforms other representations both in performance and convergence speed.