A Multi-Agent Reinforcement Learning Approach for Cooperative Air-Ground-Human Crowdsensing in Emergency Rescue

TL;DR

This work defines the HECTA problem for emergency rescue, where humans, UAVs, and UGVs cooperatively allocate sensing tasks under partial observability and strict time constraints. It introduces a Hard-Cooperative policy and models the problem as a Dec-POMDP, then presents HECTA4ER, a Centralized Training with Decentralized Execution MARL algorithm with specialized modules for feature extraction, history-aware decision making, and a global-local mixing network. The method demonstrates an average $TCR$ improvement of $18.42\%$ over baselines in simulations and validates robustness and practicality through a real-world case study. The results indicate strong potential for applying cooperative heterogeneous agents to rapid, reliable emergency response, while also outlining avenues for more realistic environment modeling and continuous-action planning.

Abstract

Mobile crowdsensing is evolving beyond traditional human-centric models by integrating heterogeneous entities like unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs). Optimizing task allocation among these diverse agents is critical, particularly in challenging emergency rescue scenarios characterized by complex environments, limited communication, and partial observability. This paper tackles the Heterogeneous-Entity Collaborative-Sensing Task Allocation (HECTA) problem specifically for emergency rescue, considering humans, UAVs, and UGVs. We introduce a novel ``Hard-Cooperative'' policy where UGVs prioritize recharging low-battery UAVs, alongside performing their sensing tasks. The primary objective is maximizing the task completion rate (TCR) under strict time constraints. We rigorously formulate this NP-hard problem as a decentralized partially observable Markov decision process (Dec-POMDP) to effectively handle sequential decision-making under uncertainty. To solve this, we propose HECTA4ER, a novel multi-agent reinforcement learning algorithm built upon a Centralized Training with Decentralized Execution architecture. HECTA4ER incorporates tailored designs, including specialized modules for complex feature extraction, utilization of action-observation history via hidden states, and a mixing network integrating global and local information, specifically addressing the challenges of partial observability. Furthermore, theoretical analysis confirms the algorithm's convergence properties. Extensive simulations demonstrate that HECTA4ER significantly outperforms baseline algorithms, achieving an average 18.42% increase in TCR. Crucially, a real-world case study validates the algorithm's effectiveness and robustness in dynamic sensing scenarios, highlighting its strong potential for practical application in emergency response.

Figures (12)

• Figure 1: An illustrative process of HECTA in emergency rescue scenarios.
• Figure 2: Grid-based modeling of the emergency rescue environment.
• Figure 3: State space. (a)-(c) show the distribution of obstacles, tasks and entities respectively; (d) represents the type of task, indicating which category of entity can perform the task; (e) depicts the remaining execution time of the task, which will gradually decrease when an agent is performing the task; (f) shows the matching status between tasks and sensing entities.
• Figure 4: Local observation space. (a) represents entity's current position; (b) shows the movable range, the cells highlighted in green, of entity $k$ within a time step; (c) depicts the power information of UAVs, including current power and power consumption per step; (d) shows entity's unique identification.
• Figure 5: Action space. (a) shows the traditional action space in route planning problem, which includes discrete movement directions. (b) represents the action space designed in our problem, denoted by the movable range of any entities at the next time step.