SE(3)-Equivariant Robot Learning and Control: A Tutorial Survey

TL;DR

This tutorial surveys SE(3)-equivariant learning and control for robotics, emphasizing how symmetry-aware architectures and geometric formulations improve sample efficiency and generalization for visual and 3D inputs. It covers foundational group theory, SE(3) Lie group/algebra, and backbones like SE(3)-equivariant GNNs and steerable CNNs, linking them to imitation and reinforcement learning. The work also develops geometric impedance control on SE(3), presenting energy-based and Lyapunov-friendly strategies that are SE(3)-equivariant, and discusses future directions including vision-to-force integration and symmetry-breaking challenges. Collectively, these insights enable robust, geometry-consistent perception, decision making, and manipulation in 3D robotic systems with unified mathematical notation.

Abstract

Recent advances in deep learning and Transformers have driven major breakthroughs in robotics by employing techniques such as imitation learning, reinforcement learning, and LLM-based multimodal perception and decision-making. However, conventional deep learning and Transformer models often struggle to process data with inherent symmetries and invariances, typically relying on large datasets or extensive data augmentation. Equivariant neural networks overcome these limitations by explicitly integrating symmetry and invariance into their architectures, leading to improved efficiency and generalization. This tutorial survey reviews a wide range of equivariant deep learning and control methods for robotics, from classic to state-of-the-art, with a focus on SE(3)-equivariant models that leverage the natural 3D rotational and translational symmetries in visual robotic manipulation and control design. Using unified mathematical notation, we begin by reviewing key concepts from group theory, along with matrix Lie groups and Lie algebras. We then introduce foundational group-equivariant neural network design and show how the group-equivariance can be obtained through their structure. Next, we discuss the applications of SE(3)-equivariant neural networks in robotics in terms of imitation learning and reinforcement learning. The SE(3)-equivariant control design is also reviewed from the perspective of geometric control. Finally, we highlight the challenges and future directions of equivariant methods in developing more robust, sample-efficient, and multi-modal real-world robotic systems.

Figures (14)

• Figure 1: Illustration of a Lie group $\mathbb{G}$ and two of its tangent spaces. The Lie algebra $\mathfrak{g} = T_{I_n}\mathbb{G}$ is the tangent space at the identity $I_n$.
• Figure 2: An illustration of a Lie group $\mathbb{G}$ and its Lie algebra $\mathfrak{g}$ of the tangent space at $\mathbbm{1}$. The Adjoint of $g$ applied on an element of $\mathfrak{g}$, i.e., $v=\frac{d}{ds} \gamma(s)|_{s=0}$ is illustrated. Note that $Ad_g v \in \mathfrak{g}.$$\Psi_g(\mathbbm{1})=g\mathbbm{1} g^{-1}=\mathbbm{1}$ implies that any curve $\gamma(s)$ through $\mathbbm{1}$ on $\mathbb{G}$ is mapped by this homomorphism $\Psi_g$ to another curve $g \gamma(s) g^{-1}$ on $\mathbb{G}$ through $\mathbbm{1}$.
• Figure 3: Summary of Lie algebra and Lie group and their adjoints that take an element of Lie algebra for $(\cdot)$.
• Figure 4: Coordinate frames $\{A\}$ and $\{B\}$ for specifying rigid motions.
• Figure 5: Transformation of wrench $F$ between coordinate frames.