Bootstrapping Reinforcement Learning with Imitation for Vision-Based Agile Flight

TL;DR

This work tackles vision-based agile quadrotor racing by bootstrapping reinforcement learning with imitation learning through a three-phase pipeline: train a state-based teacher with RL, distill a visual student via IL, and perform adaptive RL fine-tuning with an asymmetric critic to handle partial observability. The method yields robust, high-performance visuomotor policies that succeed where RL-from-scratch fails, and outperform pure IL baselines in both simulation and real-world-like hardware-in-the-loop tests. Key contributions include a performance-aware online fine-tuning strategy and the use of privileged information to stabilize learning, enabling end-to-end visual control for drone racing without explicit state estimation. The results demonstrate improved lap times, gate accuracy, and robustness across multiple tracks and input representations, highlighting the approach’s potential for real-world deployment and generalization to other visuomotor tasks.

Abstract

Learning visuomotor policies for agile quadrotor flight presents significant difficulties, primarily from inefficient policy exploration caused by high-dimensional visual inputs and the need for precise and low-latency control. To address these challenges, we propose a novel approach that combines the performance of Reinforcement Learning (RL) and the sample efficiency of Imitation Learning (IL) in the task of vision-based autonomous drone racing. While RL provides a framework for learning high-performance controllers through trial and error, it faces challenges with sample efficiency and computational demands due to the high dimensionality of visual inputs. Conversely, IL efficiently learns from visual expert demonstrations, but it remains limited by the expert's performance and state distribution. To overcome these limitations, our policy learning framework integrates the strengths of both approaches. Our framework contains three phases: training a teacher policy using RL with privileged state information, distilling it into a student policy via IL, and adaptive fine-tuning via RL. Testing in both simulated and real-world scenarios shows our approach can not only learn in scenarios where RL from scratch fails but also outperforms existing IL methods in both robustness and performance, successfully navigating a quadrotor through a race course using only visual information. Videos of the experiments are available at https://rpg.ifi.uzh.ch/bootstrap-rl-with-il/index.html.

Figures (10)

• Figure 1: Long exposure image of real-world flights shows a blue trajectory for our approach and a red one for the imitation policy. Training on the same number of samples, our approach yields a tighter trajectory, resulting in faster lap times and demonstrating superior performance and robustness.
• Figure 2: We demonstrate visuomotor policy learning in three different stages. In stage I, we train a state-based teacher policy using RL. In stage II, we use IL to learn a student distillation policy using visual inputs. In stage III, we bootstrap the actor using the student policy to fine-tune the policy through vision-based RL.
• Figure 3: Visualization of difference between the symmetric and asymmetric actor-critic learning setup.
• Figure 4: Visualization of the drone racing tracks used for the experiments, each characterized by varying levels of complexity. All the tracks maintain a consistent size scale, spanning widths from 8 meters to 16 meters.
• Figure 5: Left: Reward comparison between our approach and the other RL configurations. Ours is the only approach that is able to learn to perform the task. Right: Using a fixed sample budget, we varied the IL policy percentage for our approach on the vision-based policy for SplitS track, achieving a local maximum in deployment reward with 60$\%$ IL pretraining and 40$\%$ RL fine-tuning.