End-to-end Reinforcement Learning for Time-Optimal Quadcopter Flight

TL;DR

The paper tackles time-optimal quadcopter flight by comparing an end-to-end reinforcement learning (E2E) controller that issues direct motor commands with a residual model against an INDI-based network that outputs thrust and body-rate commands. The E2E approach achieves a 1.39 s improvement in simulation and a 0.17 s advantage in real flights over INDI, though a nontrivial sim-to-real gap remains, especially for E2E. By integrating a learned residual model for thrust and moments and an adaptive compensation mechanism, the work demonstrates the potential of end-to-end control to push latency and maneuverability limits while also outlining practical transfer challenges. The results motivate further exploration of offline RL with real flight data and additional modeling refinements to narrow the reality gap in high-speed quadcopter applications.

Abstract

Aggressive time-optimal control of quadcopters poses a significant challenge in the field of robotics. The state-of-the-art approach leverages reinforcement learning (RL) to train optimal neural policies. However, a critical hurdle is the sim-to-real gap, often addressed by employing a robust inner loop controller -an abstraction that, in theory, constrains the optimality of the trained controller, necessitating margins to counter potential disturbances. In contrast, our novel approach introduces high-speed quadcopter control using end-to-end RL (E2E) that gives direct motor commands. To bridge the reality gap, we incorporate a learned residual model and an adaptive method that can compensate for modeling errors in thrust and moments. We compare our E2E approach against a state-of-the-art network that commands thrust and body rates to an INDI inner loop controller, both in simulated and real-world flight. E2E showcases a significant 1.39-second advantage in simulation and a 0.17-second edge in real-world testing, highlighting end-to-end reinforcement learning's potential. The performance drop observed from simulation to reality shows potential for further improvement, including refining strategies to address the reality gap or exploring offline reinforcement learning with real flight data.

Figures (6)

• Figure 1: Drone racing with 2 RL Control Approaches: 1) E2E Network: Computes direct motor commands, learns compensation for unmodeled moments and thrust, trained with a learned residual model. 2) INDI Network: Computes thrust and body rate commands, uses an INDI inner loop controller, trained with first order INDI model.
• Figure 2: Flight test comparison E2E vs. INDI Net (5 Repetitions): 1) E2E outpaces INDI in the first lap (starting in hover). 2) Laps 2-6 show comparable performance. 3) E2E has a slight advantage over INDI in total track completion time. 4) Relative progress of all flights E2E leads by 0.17 seconds.
• Figure 3: Trajectory comparison E2E vs INDI: simulated runs and fastest achieved real flights. The performance gap is most significant in simulation with E2E leading by 1.39 seconds. In the real flights, this difference is reduced to 0.17 seconds.
• Figure 4: Experimental setup: The drone's position and attitude are obtained from the Optitrack motion capture system and fused with IMU data by an extended Kalman filter. Image from Ferede
• Figure 5: Modeling errors in the first order actuator delay model used for the E2E Net