Simultaneous learning of state-to-state minimum-time planning and control

TL;DR

This work tackles generalizable minimum-time UAV flight between arbitrary start and goal states by framing the problem as a reinforcement learning task and jointly learning planning and control. The authors introduce a PPO-based policy that leverages PMM-derived proxy rewards and curriculum learning to approximate the true time-optimal objective while maintaining hover stability. Key contributions include integrating PMM-based proxy rewards with a structured curriculum for scalable training and demonstrating sim-to-real transfer on outdoor flights with a low-power single-board computer. In validation, the learned policy achieves competitive flight times to NMPC baselines on PMM-generated trajectories and exhibits robust performance in real-world outdoor tests, highlighting practical impact for agile, general-purpose UAV operation.

Abstract

This paper tackles the challenge of learning a generalizable minimum-time flight policy for UAVs, capable of navigating between arbitrary start and goal states while balancing agile flight and stable hovering. Traditional approaches, particularly in autonomous drone racing, achieve impressive speeds and agility but are constrained to predefined track layouts, limiting real-world applicability. To address this, we propose a reinforcement learning-based framework that simultaneously learns state-to-state minimum-time planning and control and generalizes to arbitrary state-to-state flights. Our approach leverages Point Mass Model (PMM) trajectories as proxy rewards to approximate the true optimal flight objective and employs curriculum learning to scale the training process efficiently and to achieve generalization. We validate our method through simulation experiments, comparing it against Nonlinear Model Predictive Control (NMPC) tracking PMM-generated trajectories and conducting ablation studies to assess the impact of curriculum learning. Finally, real-world experiments confirm the robustness of our learned policy in outdoor environments, demonstrating its ability to generalize and operate on a small ARM-based single-board computer.

Figures (8)

• Figure 1: Illustration from real-world flight: quadrotor in flight using proposed learned policy.
• Figure 2: System Overview: We train the network with the trajectories generated through PMM planner as a proxy reward for minimum-time state-to-state objective. The network later does not have access to these trajectories during deployment.
• Figure 3: Comparison of mean rewards (left) and episode lengths (right) for policies trained with curriculum (green) and without curriculum (orange).
• Figure 4: Visualization of typical desired quadcopter trajectories in 3D space during simulated tests with curriculum and without curriculum. We sampled 100 initial positions. The goal conditions of all trajectories are hovering at the origin.
• Figure 5: Velocity Profiles of proposed RL controller along with the generated PMM trajectories and NMPC tracking performance. The PMM trajectories with full and reduced velocities along with NMPC performance are shown.