Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SPIRAL: Self-Play Incremental Racing Algorithm for Learning in Multi-Drone Competitions

Onur Akgün

TL;DR

SPIRAL addresses autonomous drone racing in multi-agent settings by introducing a self-play incremental learning framework, modeled as a decentralized MDP with agent $i$ observing $S_i$, taking actions $A_i$, transitioning via $P_i$, receiving rewards $R_i$, and discount factor $\gamma$, and aims to maximize $J_i(\pi_i)=E[\sum_{t=0}^{T} \gamma^t R_i(s_{i,t},a_{i,t},s'_{i,t+1})]$. It combines a high-level 4D continuous action space $(p^{x},p^{y},p^{z},\psi)$ with a PID-generated low-level control, and a composite reward that balances progress, collisions, orientation, and lap-time objectives, enabling a staged self-play curriculum. SPIRAL trains in three stages (single-drone, 1v1, and 2v2) using PPO, where opponents evolve from the agent's best past policies, and can integrate with DRL libraries; experiments on a six-gate circuit show that SPIRAL yields fastest lap times but lower reliability relative to a game-theoretic planner, demonstrating a speed-reliability trade-off. The framework is scalable and data-efficient, offering a new path toward robust, adaptive multi-agent drone racing with potential extensions toward safety constraints and explicit cooperative behavior.

Abstract

This paper introduces SPIRAL (Self-Play Incremental Racing Algorithm for Learning), a novel approach for training autonomous drones in multi-agent racing competitions. SPIRAL distinctively employs a self-play mechanism to incrementally cultivate complex racing behaviors within a challenging, dynamic environment. Through this self-play core, drones continuously compete against increasingly proficient versions of themselves, naturally escalating the difficulty of competitive interactions. This progressive learning journey guides agents from mastering fundamental flight control to executing sophisticated cooperative multi-drone racing strategies. Our method is designed for versatility, allowing integration with any state-of-the-art Deep Reinforcement Learning (DRL) algorithms within its self-play framework. Simulations demonstrate the significant advantages of SPIRAL and benchmark the performance of various DRL algorithms operating within it. Consequently, we contribute a versatile, scalable, and self-improving learning framework to the field of autonomous drone racing. SPIRAL's capacity to autonomously generate appropriate and escalating challenges through its self-play dynamic offers a promising direction for developing robust and adaptive racing strategies in multi-agent environments. This research opens new avenues for enhancing the performance and reliability of autonomous racing drones in increasingly complex and competitive scenarios.

SPIRAL: Self-Play Incremental Racing Algorithm for Learning in Multi-Drone Competitions

TL;DR

SPIRAL addresses autonomous drone racing in multi-agent settings by introducing a self-play incremental learning framework, modeled as a decentralized MDP with agent observing , taking actions , transitioning via , receiving rewards , and discount factor , and aims to maximize . It combines a high-level 4D continuous action space with a PID-generated low-level control, and a composite reward that balances progress, collisions, orientation, and lap-time objectives, enabling a staged self-play curriculum. SPIRAL trains in three stages (single-drone, 1v1, and 2v2) using PPO, where opponents evolve from the agent's best past policies, and can integrate with DRL libraries; experiments on a six-gate circuit show that SPIRAL yields fastest lap times but lower reliability relative to a game-theoretic planner, demonstrating a speed-reliability trade-off. The framework is scalable and data-efficient, offering a new path toward robust, adaptive multi-agent drone racing with potential extensions toward safety constraints and explicit cooperative behavior.

Abstract

This paper introduces SPIRAL (Self-Play Incremental Racing Algorithm for Learning), a novel approach for training autonomous drones in multi-agent racing competitions. SPIRAL distinctively employs a self-play mechanism to incrementally cultivate complex racing behaviors within a challenging, dynamic environment. Through this self-play core, drones continuously compete against increasingly proficient versions of themselves, naturally escalating the difficulty of competitive interactions. This progressive learning journey guides agents from mastering fundamental flight control to executing sophisticated cooperative multi-drone racing strategies. Our method is designed for versatility, allowing integration with any state-of-the-art Deep Reinforcement Learning (DRL) algorithms within its self-play framework. Simulations demonstrate the significant advantages of SPIRAL and benchmark the performance of various DRL algorithms operating within it. Consequently, we contribute a versatile, scalable, and self-improving learning framework to the field of autonomous drone racing. SPIRAL's capacity to autonomously generate appropriate and escalating challenges through its self-play dynamic offers a promising direction for developing robust and adaptive racing strategies in multi-agent environments. This research opens new avenues for enhancing the performance and reliability of autonomous racing drones in increasingly complex and competitive scenarios.
Paper Structure (16 sections, 3 equations, 3 figures, 2 tables)

This paper contains 16 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Problem Formulation: Multi-Drone Racing as a Markov Decision Process
  5. Simulation Environment and Drone Control Model
  6. SPIRAL: Self-Play Incremental Racing Algorithm for Learning
  7. State Space Representation ($S_i$)
  8. Action Space ($A_i$)
  9. Reward Function ($R_i$)
  10. Staged Training Protocol and Self-Play Dynamics
  11. Results and Discussion
  12. Experimental Setup
  13. Performance Metrics
  14. Experimental Results
  15. Discussion of Results
  16. ...and 1 more sections

Figures (3)

  • Figure 1: The SPIRAL (Self-Play Incremental Racing Algorithm for Learning) training framework consists of three progressive stages. Stage 1 establishes fundamental flight control capabilities in isolation. Stages 2 and 3 employ self-play training loops where agents continuously improve by racing against their previous best policies, with Stage 2 focusing on 1v1 competition and Stage 3 extending to 2v2 team dynamics. The self-play mechanism (shown in dashed boxes) enables continuous improvement without requiring external expert demonstrations.
  • Figure 2: Visualization of the 2v2 drone racing environment. The race track consists of six gates arranged in a circuit on a checkered floor. Four drones (two red and two white) are positioned on the track, representing the 2v2 team formation used in our experiments. The blue axis indicator provides spatial reference.
  • Figure 3: Speed vs. Reliability Trade-off. This plot visualizes the performance of each method in both the 1v1 (circles) and 2v2 (squares) scenarios. The x-axis represents the average lap time (lower is better), and the y-axis represents the success ratio (higher is better). The top-left corner is the "Ideal Region," representing both high speed and high reliability.