Leveraging Symmetry to Accelerate Learning of Trajectory Tracking Controllers for Free-Flying Robotic Systems
Jake Welde, Nishanth Rao, Pratik Kunapuli, Dinesh Jayaraman, Vijay Kumar
TL;DR
This work addresses the data inefficiency of reinforcement learning for tracking controllers in free-flying robots by exploiting continuous Lie group symmetries. It formulates trajectory tracking as a stationary continuous MDP and proves that symmetry induces a quotient MDP via an MDP homomorphism, enabling policy lifting to the original system with preserved optimality and Q-values. The authors derive explicit quotient constructions for Particle, Astrobee, and Quadrotor, demonstrating accelerated training and lower tracking error through symmetry-aware learning, including zero-shot generalization to planned trajectories. The framework provides a principled method to reduce problem dimensionality while maintaining performance, with practical impact on efficient RL for complex robotic systems.
Abstract
Tracking controllers enable robotic systems to accurately follow planned reference trajectories. In particular, reinforcement learning (RL) has shown promise in the synthesis of controllers for systems with complex dynamics and modest online compute budgets. However, the poor sample efficiency of RL and the challenges of reward design make training slow and sometimes unstable, especially for high-dimensional systems. In this work, we leverage the inherent Lie group symmetries of robotic systems with a floating base to mitigate these challenges when learning tracking controllers. We model a general tracking problem as a Markov decision process (MDP) that captures the evolution of both the physical and reference states. Next, we prove that symmetry in the underlying dynamics and running costs leads to an MDP homomorphism, a mapping that allows a policy trained on a lower-dimensional "quotient" MDP to be lifted to an optimal tracking controller for the original system. We compare this symmetry-informed approach to an unstructured baseline, using Proximal Policy Optimization (PPO) to learn tracking controllers for three systems: the Particle (a forced point mass), the Astrobee (a fullyactuated space robot), and the Quadrotor (an underactuated system). Results show that a symmetry-aware approach both accelerates training and reduces tracking error at convergence.