Constrained Reinforcement Learning using Distributional Representation for Trustworthy Quadrotor UAV Tracking Control
Yanran Wang, David Boyle
TL;DR
This work tackles accurate, trustworthy quadrotor trajectory tracking under highly uncertain aerodynamic disturbances by integrating ConsDRED, a quantile-based constrained distributional RL disturbance estimator, with a SADF-SMPC controller. The CDMDP formulation and distributional TD learning ensure convergence at a sublinear rate, with global policy improvement guaranteed by contraction in the Wasserstein metric and a Lipschitz, ISS-stable closed-loop system under non-zero-mean disturbances. Empirical results in simulation and real flights demonstrate that ConsDRED-SMPC outperforms state-of-the-art CRL and SMPC-based methods, achieving up to a 70% reduction in RMSE tracking error and substantial gains in constrained returns, while remaining robust to hyperparameter settings. The framework leverages wind estimation (VID-Fusion) and preserves convexity via SADF, offering a practical, scalable approach for trustworthy learning-based quadrotor tracking in complex environments. Key contributions include the global convergence guarantees, ISS stability, and demonstrated practicality across simulated and real-world scenarios with non-zero-mean disturbances. All mathematical notation is presented with appropriate delimiters, e.g., $\Theta(1/\sqrt{T})$, $m$, $H$, and distributional operators.
Abstract
Simultaneously accurate and reliable tracking control for quadrotors in complex dynamic environments is challenging. As aerodynamics derived from drag forces and moment variations are chaotic and difficult to precisely identify, most current quadrotor tracking systems treat them as simple `disturbances' in conventional control approaches. We propose a novel, interpretable trajectory tracker integrating a Distributional Reinforcement Learning disturbance estimator for unknown aerodynamic effects with a Stochastic Model Predictive Controller (SMPC). The proposed estimator `Constrained Distributional Reinforced disturbance estimator' (ConsDRED) accurately identifies uncertainties between true and estimated values of aerodynamic effects. Simplified Affine Disturbance Feedback is used for control parameterization to guarantee convexity, which we then integrate with a SMPC. We theoretically guarantee that ConsDRED achieves at least an optimal global convergence rate and a certain sublinear rate if constraints are violated with an error decreases as the width and the layer of neural network increase. To demonstrate practicality, we show convergent training in simulation and real-world experiments, and empirically verify that ConsDRED is less sensitive to hyperparameter settings compared with canonical constrained RL approaches. We demonstrate our system improves accumulative tracking errors by at least 70% compared with the recent art. Importantly, the proposed framework, ConsDRED-SMPC, balances the tradeoff between pursuing high performance and obeying conservative constraints for practical implementations.