Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Curriculum-Based Iterative Self-Play for Scalable Multi-Drone Racing

Onur Akgün

TL;DR

CRUISE tackles scalable coordination for $N$ autonomous drones in high-speed racing by marrying curriculum learning with iterative self-play within a PPO framework. The approach starts with a five-stage curriculum to build single-agent competence, then engages in iterative self-play against frozen, increasingly capable opponents to foster robust competitive strategies. Ablation studies show the curriculum is essential for sample efficiency and the emergence of sophisticated tactics that balance speed and safety. Empirical results in high-fidelity drone simulations demonstrate CRUISE markedly outperforming a game-theoretic planner and a vanilla RL baseline, with code and pretrained models released for reproducibility.

Abstract

The coordination of multiple autonomous agents in high-speed, competitive environments represents a significant engineering challenge. This paper presents CRUISE (Curriculum-Based Iterative Self-Play for Scalable Multi-Drone Racing), a reinforcement learning framework designed to solve this challenge in the demanding domain of multi-drone racing. CRUISE overcomes key scalability limitations by synergistically combining a progressive difficulty curriculum with an efficient self-play mechanism to foster robust competitive behaviors. Validated in high-fidelity simulation with realistic quadrotor dynamics, the resulting policies significantly outperform both a standard reinforcement learning baseline and a state-of-the-art game-theoretic planner. CRUISE achieves nearly double the planner's mean racing speed, maintains high success rates, and demonstrates robust scalability as agent density increases. Ablation studies confirm that the curriculum structure is the critical component for this performance leap. By providing a scalable and effective training methodology, CRUISE advances the development of autonomous systems for dynamic, competitive tasks and serves as a blueprint for future real-world deployment.

Curriculum-Based Iterative Self-Play for Scalable Multi-Drone Racing

TL;DR

CRUISE tackles scalable coordination for autonomous drones in high-speed racing by marrying curriculum learning with iterative self-play within a PPO framework. The approach starts with a five-stage curriculum to build single-agent competence, then engages in iterative self-play against frozen, increasingly capable opponents to foster robust competitive strategies. Ablation studies show the curriculum is essential for sample efficiency and the emergence of sophisticated tactics that balance speed and safety. Empirical results in high-fidelity drone simulations demonstrate CRUISE markedly outperforming a game-theoretic planner and a vanilla RL baseline, with code and pretrained models released for reproducibility.

Abstract

The coordination of multiple autonomous agents in high-speed, competitive environments represents a significant engineering challenge. This paper presents CRUISE (Curriculum-Based Iterative Self-Play for Scalable Multi-Drone Racing), a reinforcement learning framework designed to solve this challenge in the demanding domain of multi-drone racing. CRUISE overcomes key scalability limitations by synergistically combining a progressive difficulty curriculum with an efficient self-play mechanism to foster robust competitive behaviors. Validated in high-fidelity simulation with realistic quadrotor dynamics, the resulting policies significantly outperform both a standard reinforcement learning baseline and a state-of-the-art game-theoretic planner. CRUISE achieves nearly double the planner's mean racing speed, maintains high success rates, and demonstrates robust scalability as agent density increases. Ablation studies confirm that the curriculum structure is the critical component for this performance leap. By providing a scalable and effective training methodology, CRUISE advances the development of autonomous systems for dynamic, competitive tasks and serves as a blueprint for future real-world deployment.
Paper Structure (27 sections, 17 equations, 5 figures, 4 tables, 1 algorithm)

This paper contains 27 sections, 17 equations, 5 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Drone Dynamics and Controller Design
  5. Notation and Dynamics
  6. Hierarchical Controller
  7. Velocity Tracking Controller (Outer Loop)
  8. Low-Level Controller (Inner Loop)
  9. Racing Agent Design and Training
  10. State and Action Space
  11. Action Space
  12. Reward Function
  13. Curriculum Learning Implementation
  14. Curriculum Design and Progression
  15. Self-Play Methodology
  16. ...and 12 more sections

Figures (5)

  • Figure 1: Quadrotor reference frames and rotor configuration. The world frame axes ($x_W, y_W, z_W$) are fixed, with $z_W$ pointing upward. The body frame ($x_B, y_B, z_B$) is attached to the drone's center of mass with $x_B$ pointing forward. Blue arrows indicate thrust direction, while curved arrows show rotor rotation direction.
  • Figure 2: Representative trajectories of four CRUISE-trained drones navigating the Ring Track. The track features five gates (black rectangles) in a circular layout with alternating heights. Distinct colors show individual drone paths, illustrating coordinated high-speed navigation.
  • Figure 3: Representative trajectories of four CRUISE-trained drones on the more complex Figure-Eight Track. The six gates form intersecting loops, requiring intricate coordination at crossing points.
  • Figure 4: Ablation study on the Ring Track: Mean velocity (m/s) vs. curriculum stage for $N=2, 3, 4$ drones. Error bars indicate std dev (100 trials). Inset shows track layout. Increasing velocity across stages demonstrates the curriculum's benefit.
  • Figure 5: Ablation study on the Figure-Eight Track: Mean velocity (m/s) vs. curriculum stage for $N=2, 3, 4$ drones. Error bars show std dev (100 trials). Inset shows track layout. Performance increases consistently with curriculum stage.