Dream to Fly: Model-Based Reinforcement Learning for Vision-Based Drone Flight

TL;DR

This work tackles vision-based autonomous drone racing by learning end-to-end visuomotor policies that map raw camera pixels directly to CTBR commands without explicit state estimation. It adopts a model-based reinforcement learning approach (DreamerV3) to learn a world model and an actor-critic policy from pixels, achieving data-efficient training and successful transfer to real quadrotors. The authors demonstrate that DreamerV3 outperforms a model-free baseline (PPO), exhibits emergent perception-aware behavior, and maintains a small sim-to-real gap when deployed in a real-world Figure 8 track. Overall, the study advances vision-based autonomous flight by showing that pixel-to-command control is feasible with model-based RL, enabling robust, perception-rich decision making in real-world robotics.

Abstract

Autonomous drone racing has risen as a challenging robotic benchmark for testing the limits of learning, perception, planning, and control. Expert human pilots are able to agilely fly a drone through a race track by mapping the real-time feed from a single onboard camera directly to control commands. Recent works in autonomous drone racing attempting direct pixel-to-commands control policies (without explicit state estimation) have relied on either intermediate representations that simplify the observation space or performed extensive bootstrapping using Imitation Learning (IL). This paper introduces an approach that learns policies from scratch, allowing a quadrotor to autonomously navigate a race track by directly mapping raw onboard camera pixels to control commands, just as human pilots do. By leveraging model-based reinforcement learning~(RL) - specifically DreamerV3 - we train visuomotor policies capable of agile flight through a race track using only raw pixel observations. While model-free RL methods such as PPO struggle to learn under these conditions, DreamerV3 efficiently acquires complex visuomotor behaviors. Moreover, because our policies learn directly from pixel inputs, the perception-aware reward term employed in previous RL approaches to guide the training process is no longer needed. Our experiments demonstrate in both simulation and real-world flight how the proposed approach can be deployed on agile quadrotors. This approach advances the frontier of vision-based autonomous flight and shows that model-based RL is a promising direction for real-world robotics.

Figures (7)

• Figure 1: Real-world deployment of our DreamerV3 policy in the Figure 8 track. During training, the agent learns a world model from interactions with the environment. At the same time, the actor-critic policy is trained by sampling the predictions of the world model, also called imagination. The onboard images consumed by the network are shown in red.
• Figure 2: The process begins with data collection in the real environment using the current policy, storing experiences in a replay buffer. This buffer is used to train the world model components: the encoder, decoder, RSSM, dynamics model, reward model, and continue predictor (Section \ref{['sec:dreamerv3']}). Subsequently, an actor-critic policy is trained within the learned world model to maximize expected (imagined) returns. This updated policy is then used to collect new data, restarting the loop.
• Figure 3: Reward evolution by number of steps for three different tracks: Circle track, Kidney Track and Figure 8 track. The training performance of DreamerV3 is shown in blue, and for PPO in orange. We show that while PPO is not able to achieve any considerable training in 10 million environment interactions, DreamerV3 is able to train to convergence for the three tracks.
• Figure 4: Comparison of real observations and imagined observations for the Figure 8 track. Imagined observations are observations that are reconstructed by the world model. The figure shows the reconstructed observations after 0.4M steps (early stage training), 1M timesteps (mid stage training), and after 10M steps (training convergence). One can observe how the reconstruction gets better and better as the training evolves.
• Figure 5: Ablation study on the perception aware behaviour of our policies.Top: DreamerV3 policy trained on pixel observations in an environment where the only rendered gates are the actual gates. As indicated by the black arrows (representing camera direction), the platform predominantly focuses its attention on the next gate. Bottom: We introduce two additional gates to the rendering engine (marked in red color). These gates are not required to be traversed but serve as a valuable source of information for platform localization. Consequently, the policy's behavior shifts, and the platform now distributes its attention more evenly across both the actual and the extra gates.