Reinforcement Learning-based Fault-Tolerant Control for Quadrotor with Online Transformer Adaptation

TL;DR

A hybrid RL-based fault-tolerant control framework for quadrotors combines a nominal cascaded PID controller, an RL policy, and a transformer-based online adaptation module to infer latent dynamics without retraining. The framework uses a two-phase training pipeline and a teacher-student adaptation module to achieve robust online inference in unseen configurations. In PyBullet hover tasks with actuator loss up to 36%, it achieves 95% success and RMSE of 0.129 m, outperforming RL-RMA and PID baselines, and demonstrates robustness across untrained dynamics and configurations. The approach enables practical fault management in dynamic environments by providing rapid, adaptive control without privileged information or re-training.

Abstract

Multirotors play a significant role in diverse field robotics applications but remain highly susceptible to actuator failures, leading to rapid instability and compromised mission reliability. While various fault-tolerant control (FTC) strategies using reinforcement learning (RL) have been widely explored, most previous approaches require prior knowledge of the multirotor model or struggle to adapt to new configurations. To address these limitations, we propose a novel hybrid RL-based FTC framework integrated with a transformer-based online adaptation module. Our framework leverages a transformer architecture to infer latent representations in real time, enabling adaptation to previously unseen system models without retraining. We evaluate our method in a PyBullet simulation under loss-of-effectiveness actuator faults, achieving a 95% success rate and a positional root mean square error (RMSE) of 0.129 m, outperforming existing adaptation methods with 86% success and an RMSE of 0.153 m. Further evaluations on quadrotors with varying configurations confirm the robustness of our framework across untrained dynamics. These results demonstrate the potential of our framework to enhance the adaptability and reliability of multirotors, enabling efficient fault management in dynamic and uncertain environments. Website is available at http://00dhkim.me/paper/rl-ftc

Figures (4)

• Figure 1: The overview of 2-phase training and deployment. Policy Training: In phase 1, the environmental factor encoder $\mu$ and policy $\pi$ are jointly trained. $\mu$ encodes privileged factors into a latent vector $z_t$. The policy then uses $z_t$ together with state $x_t$ and previous action $u_{t-1}$ to generate the control command $u^\text{rl}_t$. The command is added to the nominal controller's command $u^\text{nominal}_t$ and applied to the dynamics model. Adaptation Module Training: The adaptation module $h$ learns to infer $\hat{z}_t$ without access to privileged state $\xi_t$, through supervised learning to minimize the error between $\hat{z}_t$ and ground truth $z_t$. Training data is collected using the frozen policy $\pi$. Deployment and Testing: In deployment, both $\pi$ and $h$ are frozen. $h$ infers $\hat{z}_t$ from trajectory, which $\pi$ uses to produce control commands. This enables adaptability to unseen dynamics while achieving FTC.
• Figure 2: State comparison between our method and the baselines for each quadrotor under 30% LoE of $F_2$. The black dashed lines are the target state.
• Figure 3: State comparison between our method and the baselines for the UAV1 model with 100 fault cases. Colored solid lines are successful cases, black solid lines are failure cases, and red dotted lines are the target state.
• Figure 4: Position RMSE and success/failure results between our method and the baselines for the UAV1 model with 100 fault cases. The cases where the RMSE exceeds 0.7m are noted on the boundary.