Deep Reinforcement Learning Policies for Underactuated Satellite Attitude Control

TL;DR

This work formulates autonomous satellite attitude control under actuator failure as a model-free reinforcement learning problem and trains continuous-action policies using a custom PPO. It builds ten controllers (one nominal and nine underactuated, axis-specialized) in simulation, then validates them on hardware-in-the-loop with a representative small-satellite platform, demonstrating robust sim-to-hardware transfer. The key findings show fast convergence to precise pointing (target error near $0.01$ rad) even under underactuation and communication delays, indicating RL can yield resilient attitude control without full actuator redundancy. The study suggests RL-based approaches can adapt to different inertial properties and supports progressive flight-testing to establish practical viability for deep-space small-satellite missions.

Abstract

Autonomy is a key challenge for future space exploration endeavours. Deep Reinforcement Learning holds the promises for developing agents able to learn complex behaviours simply by interacting with their environment. This paper investigates the use of Reinforcement Learning for the satellite attitude control problem, namely the angular reorientation of a spacecraft with respect to an in- ertial frame of reference. In the proposed approach, a set of control policies are implemented as neural networks trained with a custom version of the Proximal Policy Optimization algorithm to maneuver a small satellite from a random starting angle to a given pointing target. In particular, we address the problem for two working conditions: the nominal case, in which all the actuators (a set of 3 reac- tion wheels) are working properly, and the underactuated case, where an actuator failure is simulated randomly along with one of the axes. We show that the agents learn to effectively perform large-angle slew maneuvers with fast convergence and industry-standard pointing accuracy. Furthermore, we test the proposed method on representative hardware, showing that by taking adequate measures controllers trained in simulation can perform well in real systems.

Figures (4)

• Figure 1: Here the three phases of the system implementation are presented: the training of the agents which sees the interaction of the agent with the python satellite's attitude simulator using the actor-critic method, then the evaluation performed using the simulated environment and the actor network used in a deterministic fashion, finally the Hardware-in-the-Loop testing where the simulator is replaced by the ground model components of the satellite (the OBC and ADCS) along with the Real Dynamic Processor (RDP) used to propagate the attitude states.
• Figure 2: Agents control history for the nominal case with average and variance performances in blue line and blue area, and absolute worst case performances highlighted by the red area. The statistics are obtained over 10000 runs and show how both the average and the worst case trajectory converge to the 0.01 rad threshold in less than 100 seconds.
• Figure 3: Agents control history for the off-nominal conditions, tabulated in \ref{['tab-failed']}, with average and variance performances in blue line and blue area, and absolute worst case performances highlighted by the red area. The statistics are obtained over 10000 runs for each of the nine controllers.
• Figure 4: Agent control history for two attitude maneuvers performed using the HiL setup. The quaternion, the body rates, the RWs speeds and the commanded torques are shown in (a) and (b) for a nominal and an underactuated (with failure on the X axis) maneuver. In (c) and (d) their respective trajectories are plotted over the spherical unit quaternion.