RL-based Control of UAS Subject to Significant Disturbance

TL;DR

This work tackles proactive disturbance rejection for quadrotor UAS subjected to predictable impulse recoil, such as from a water cannon. It introduces a reinforcement learning framework that augments the policy's observations with a warning trigger signal, enabling preemptive stabilization before the disturbance occurs. Through PPO-based training in a high-fidelity simulator with domain randomization, three policies are compared: a nominal policy, an impulse-aware policy without the trigger, and a predictive policy that utilizes the trigger. The predictive policy demonstrates superior performance by initiating preemptive corrections, reducing position error and control effort relative to reactive or baselined approaches. The results highlight the potential of predictive cues to enhance RL-based control in disturbance-prone aerial systems and suggest directions for real-world validation and generalization to other platforms and disturbance types.

Abstract

This paper proposes a Reinforcement Learning (RL)-based control framework for position and attitude control of an Unmanned Aerial System (UAS) subjected to significant disturbance that can be associated with an uncertain trigger signal. The proposed method learns the relationship between the trigger signal and disturbance force, enabling the system to anticipate and counteract the impending disturbances before they occur. We train and evaluate three policies: a baseline policy trained without exposure to the disturbance, a reactive policy trained with the disturbance but without the trigger signal, and a predictive policy that incorporates the trigger signal as an observation and is exposed to the disturbance during training. Our simulation results show that the predictive policy outperforms the other policies by minimizing position deviations through a proactive correction maneuver. This work highlights the potential of integrating predictive cues into RL frameworks to improve UAS performance.

Figures (11)

• Figure 1: IFEX system firing a high-pressure water burst, illustrating the significant recoil force experienced by the UAS.
• Figure 2: Schematic representation of a quadrotor with its reference frames, and the disturbance force $\mathbf{f}_d$ which is applied along the $\hat{\mathbf{x}}_B$ axis.
• Figure 3: Temporal relation between the impulse disturbance force $\mathbf{f}_d$ and the trigger signal $o_t$. Note that the amplitude of the disturbance $\mathbf{f}_d$ is normalized for visualization purposes only.
• Figure 4: Block diagram of the proposed control framework.
• Figure 5: Overview of the average return comparison of the three policies.