Mastering Multi-Drone Volleyball through Hierarchical Co-Self-Play Reinforcement Learning

TL;DR

Hierarchical Co-Self-Play (HCSP) is proposed, a hierarchical reinforcement learning framework that separates centralized high-level strategic decision-making from decentralized low-level motion control and leads to emergent team behaviors such as role switching and coordinated formations, demonstrating the effectiveness of the hierarchical design and training scheme.

Abstract

In this paper, we tackle the problem of learning to play 3v3 multi-drone volleyball, a new embodied competitive task that requires both high-level strategic coordination and low-level agile control. The task is turn-based, multi-agent, and physically grounded, posing significant challenges due to its long-horizon dependencies, tight inter-agent coupling, and the underactuated dynamics of quadrotors. To address this, we propose Hierarchical Co-Self-Play (HCSP), a hierarchical reinforcement learning framework that separates centralized high-level strategic decision-making from decentralized low-level motion control. We design a three-stage population-based training pipeline to enable both strategy and skill to emerge from scratch without expert demonstrations: (I) training diverse low-level skills, (II) learning high-level strategy via self-play with fixed low-level skills, and (III) joint fine-tuning through co-self-play. Experiments show that HCSP achieves superior performance, outperforming non-hierarchical self-play and rule-based hierarchical baselines with an average 82.9% win rate and a 71.5% win rate against the two-stage variant. Moreover, co-self-play leads to emergent team behaviors such as role switching and coordinated formations, demonstrating the effectiveness of our hierarchical design and training scheme. The project page is at https://hi-co-self-play.github.io.

Figures (12)

• Figure 1: HCSP architecture: an event‐driven high‐level strategy handles strategic decisions, while multiple low‐level skills manage continuous control. Each drone $i$ runs its low‐level skill at 50Hz, taking local observation $o_i^L$ and high‐level tactical parameters to produce continuous action $a_i^L$. The high‐level strategy activates only on discrete events (racket strike or ball crossing the net), observes $o^H$, and outputs $a^H$ to choose each drone’s skill and supply tactical information. It is implemented as a shared MLP with three output heads, one per drone.
• Figure 2: Illustrations of the high-level strategy pretraining stage (Stage II).
• Figure 3: Illustration of the co-self-play stage (Stage III).
• Figure 4: Experiment results. (a) HCSP performance against baseline policies. (b) Cross-play win rate heatmap among ten randomly picked policies. (c) Ablation study on policy chaining in Stage I: policy chaining enables successful transition from Attack to Hover, whereas single-policy fails.
• Figure 5: Sequence of six temporally sampled frames illustrating an emergent team behavior. (a) The opponent team passes the ball to the setter. (b) The opponent team sets the ball to the attacker. (c) The opponent team attacks the ball towards our side. (d) Our team passes the ball to the setter. (e) Our team performs a "dump" shot, sending the ball directly onto the opponent’s court. (f) The opponent team fails to return the ball, so our team scores the point.