SkyDreamer: Interpretable End-to-End Vision-Based Drone Racing with Model-Based Reinforcement Learning

TL;DR

SkyDreamer introduces a first end-to-end vision-based autonomous drone racing policy that maps pixel inputs directly to motor commands, learned within an informed POMDP framework and guided by privileged information during training to yield an interpretable world model. Building on DreamerV3, the approach uses an RSSM that decodes privileged states and parameters, enabling latent imagination for actor-critic learning and on-board execution without inner-loop controllers or external aids. A flight-plan logic and robust visual augmentation (CycleGAN/StochGAN) bridge the reality gap, enabling non-trivial sim-to-real transfer even with low-quality segmentation masks. Real-world experiments across small and large tracks demonstrate up to 21 m/s speeds and 6 g accelerations, on-board operation, and adaptive behavior to battery degradation, highlighting SkyDreamer’s potential for robust, high-speed, end-to-end vision-based drone racing. While parameter estimation drifts and computational demands persist, the work marks a significant step toward generalizable, interpretable, ultra-responsive autonomous flight in real-world settings.

Abstract

Autonomous drone racing (ADR) systems have recently achieved champion-level performance, yet remain highly specific to drone racing. While end-to-end vision-based methods promise broader applicability, no system to date simultaneously achieves full sim-to-real transfer, onboard execution, and champion-level performance. In this work, we present SkyDreamer, to the best of our knowledge, the first end-to-end vision-based ADR policy that maps directly from pixel-level representations to motor commands. SkyDreamer builds on informed Dreamer, a model-based reinforcement learning approach where the world model decodes to privileged information only available during training. By extending this concept to end-to-end vision-based ADR, the world model effectively functions as an implicit state and parameter estimator, greatly improving interpretability. SkyDreamer runs fully onboard without external aid, resolves visual ambiguities by tracking progress using the state decoded from the world model's hidden state, and requires no extrinsic camera calibration, enabling rapid deployment across different drones without retraining. Real-world experiments show that SkyDreamer achieves robust, high-speed flight, executing tight maneuvers such as an inverted loop, a split-S and a ladder, reaching speeds of up to 21 m/s and accelerations of up to 6 g. It further demonstrates a non-trivial visual sim-to-real transfer by operating on poor-quality segmentation masks, and exhibits robustness to battery depletion by accurately estimating the maximum attainable motor RPM and adjusting its flight path in real-time. These results highlight SkyDreamer's adaptability to important aspects of the reality gap, bringing robustness while still achieving extremely high-speed, agile flight.

Figures (9)

• Figure 1: Real-world inverted loop track with MAVLab gates. SkyDreamer flies agile maneuvers such as an inverted loop in the real world without external aid (top), operates end-to-end on poor-quality segmentation masks obtained from onboard images (shown in red, bottom-right), and directly controls the physical platform by outputting low-level motor commands. In a flight area confined to $6 \times 6 \, \text{m}$, it reaches speeds of up to $13 \, \text{m/s}$ and accelerations of up to $6 \, \text{g}$. The racing drone is identical to the one used in the A2RL x DCL Autonomous Drone Championship 2025 aibeatshumandeWagter2025 (bottom-left).
• Figure 2: Coordinate systems. A quadcopter pitched forward -- shown as a cross with four circles -- with a camera pitched upward, is shown flying toward the gate at the bottom left. Subscripts indicate the reference frames: $w$ - world frame, $g$ - gate frame, $b$ - body frame, and $c$ - camera frame. Image not to scale, the axes of the frames are not necessarily aligned.
• Figure 3: Comparison of standard DreamerV3 and SkyDreamer across different stages. The world model encodes segmentation masks and measurements such as motor RPMs, body rates, and the flight plan vector into discrete representations $z_t$. The next hidden state $h_{t+1}$ is predicted from the previous hidden state $h_t$, action $a_t$ and discrete representation $z_t$ using the sequence model. The inputs $\hat{x}_t$ are reconstructed to guide the hidden representations. In SkyDreamer, the privileged observations, including ground-truth state, camera and dynamical parameters, are decoded as $\mathbf{o}_t^+$. During actor-critic learning, imagined rollouts are generated solely using the dynamics predictor without feedback from the environment. Images inspired by Hafner et al.dreamerv3_nature.
• Figure 4: Simulated ladder inverted loop track flown by SkyDreamer. The flight begins by passing through the first gate, followed by a tight ladder maneuver and flying over it. SkyDreamer then flies over the second gate, performs a maneuver similar to a split-S through it, and flies back over the gate, completing the inverted loop. The lap is completed by passing through the first gate again. In the left image, the black lines indicate ground-truth trajectory, while the colored lines show SkyDreamer's position and velocity estimates, with color denoting speed. The black blocks mark the gate locations with exaggerated thickness. The black arrows indicate the camera principal axis direction, which show that SkyDreamer naturally orients its camera toward the gates: an emergent behavior achieved without any explicit perception reward. The right image presents a 3D render of one lap flown by SkyDreamer, where the colored line represents the ground-truth trajectory and corresponding speed.
• Figure 5: Selected SkyDreamer parameter, extrinsic, and state estimation over time for the ladder inverted loop track with orange gates. The left half of each plot shows the first 150 steps, highlighting the speed of convergence, while the right half shows the remaining 1850 steps, illustrating any drift over time. The red distribution represents the baseline, computed using the theoretical standard deviation of a uniform distribution representing the distribution of the training ranges in \ref{['tab:randomization_ranges']}. The green distribution shows SkyDreamer’s estimation over time, obtained by aggregating results from 20 differently initialized drones that completed a successful flight over 2000 timesteps each, following the evaluation parameter ranges from \ref{['tab:randomization_ranges']}. Distributions such as those for $k_x$ align with the baseline distribution, indicating that SkyDreamer failed to estimate these parameters. Other distributions are narrower, showing that SkyDreamer successfully estimated them on the fly. Parameters like $\omega_{\max}$ and $k_w$ converge quickly and remain stable over time, while parameters like $k_{p1}$ converge more slowly and show gradual drift over time.