Dynamic Sight Range Selection in Multi-Agent Reinforcement Learning

TL;DR

This work addresses the sight range dilemma in multi-agent reinforcement learning by introducing Dynamic Sight Range Selection (DSR), a meta-controller that uses a sliding-window UCB strategy to dynamically choose each episode's sight range without requiring global information. The observation function is augmented to depend on the chosen sight range, enabling hierarchical optimization where the meta-controller and the MARL agents learn jointly and asynchronously. Across LBF, RWARE, and SMAC, and for multiple MARL algorithms including QMIX and MAPPO, DSR yields performance gains, accelerates training, and provides interpretability by revealing the selected sight ranges. The approach is hardware- and algorithm-agnostic, easily integrating with existing MARL frameworks and offering practical implications for sensor design and future extensions to heterogeneous agents and continuous observation spaces.

Abstract

Multi-agent reinforcement Learning (MARL) is often challenged by the sight range dilemma, where agents either receive insufficient or excessive information from their environment. In this paper, we propose a novel method, called Dynamic Sight Range Selection (DSR), to address this issue. DSR utilizes an Upper Confidence Bound (UCB) algorithm and dynamically adjusts the sight range during training. Experiment results show several advantages of using DSR. First, we demonstrate using DSR achieves better performance in three common MARL environments, including Level-Based Foraging (LBF), Multi-Robot Warehouse (RWARE), and StarCraft Multi-Agent Challenge (SMAC). Second, our results show that DSR consistently improves performance across multiple MARL algorithms, including QMIX and MAPPO. Third, DSR offers suitable sight ranges for different training steps, thereby accelerating the training process. Finally, DSR provides additional interpretability by indicating the optimal sight range used during training. Unlike existing methods that rely on global information or communication mechanisms, our approach operates solely based on the individual sight ranges of agents. This approach offers a practical and efficient solution to the sight range dilemma, making it broadly applicable to real-world complex environments.

Figures (43)

• Figure 1:
• Figure 2:
• Figure 3:
• Figure 5: Overview of the Dynamic Sight Range Selection framework. The meta-controller (left) dynamically selects the current optimal sight range $d_e^*$ using the sliding-window UCB based on the episode return $r_e$. The selected sight range $d_e^*$ is used in the MARL training (right), where agents interact with the environment, receiving observations within the selected sight range.
• Figure 6: Level-Based Foraging (LBF)