Distributed Area Coverage with High Altitude Balloons Using Multi-Agent Reinforcement Learning

TL;DR

This work tackles distributed area coverage by coordinating multiple high-altitude balloons (HABs) under complex stratospheric winds. It extends the RLHAB simulation environment to support cooperative multi-agent reinforcement learning and applies QMIX within a Centralized Training with Decentralized Execution framework, augmented by a rich observation and a hierarchical reward structure to address credit assignment. The results show QMIX can achieve performance on par with deterministic Voronoi-Lloyd baselines for simple coverage tasks, while enabling adaptive, wind-aware strategies that are scalable to more complex missions. The study demonstrates a concrete, learning-based foundation for small-team HAB coordination with potential to scale to heterogeneous, multi-objective deployments where deterministic methods become intractable.

Abstract

High Altitude Balloons (HABs) can leverage stratospheric wind layers for limited horizontal control, enabling applications in reconnaissance, environmental monitoring, and communications networks. Existing multi-agent HAB coordination approaches use deterministic methods like Voronoi partitioning and extremum seeking control for large global constellations, which perform poorly for smaller teams and localized missions. While single-agent HAB control using reinforcement learning has been demonstrated on HABs, coordinated multi-agent reinforcement learning (MARL) has not yet been investigated. This work presents the first systematic application of multi-agent reinforcement learning (MARL) to HAB coordination for distributed area coverage. We extend our previously developed reinforcement learning simulation environment (RLHAB) to support cooperative multi-agent learning, enabling multiple agents to operate simultaneously in realistic atmospheric conditions. We adapt QMIX for HAB area coverage coordination, leveraging Centralized Training with Decentralized Execution to address atmospheric vehicle coordination challenges. Our approach employs specialized observation spaces providing individual state, environmental context, and teammate data, with hierarchical rewards prioritizing coverage while encouraging spatial distribution. We demonstrate that QMIX achieves similar performance to the theoretically optimal geometric deterministic method for distributed area coverage, validating the MARL approach and providing a foundation for more complex autonomous multi-HAB missions where deterministic methods become intractable.

Figures (9)

• Figure 1: RLHAB v.2 simulation environment extended to support multi-agent dynamics and control with different forecast and ground truth wind flow fields.
• Figure 2: Voronoi partitions, centroid waypoints, and HAB trajectories changing over time.
• Figure 3: Optimized agent separation with Voronoi partitions after 20 iterations of Lloyd's relaxation, constrained to a circular boundary for 4, 5, 6, and 12 agents.
• Figure 4: QMIX Learning Curves for Reward, Separation Ratio, and Mean Group TWR over 7000 episodes (approximately 20 million timesteps) for 4 and 6 agents.
• Figure 5: A sampling of final trajectories and coverage maps between the Voronoi Baseline Controller and QMIX for the same forecast and starting positions. The colored coverage heatmap is capped off at episode length.