SMAUG: A Sliding Multidimensional Task Window-Based MARL Framework for Adaptive Real-Time Subtask Recognition

TL;DR

SMAUG addresses real-time subtask recognition in cooperative MARL under partial observability by introducing a sliding multidimensional task window, an inference network for future trajectory prediction, and intrinsic motivation rewards to encourage subtask exploration and diversity. The framework combines subtask encodings with a QMIX-style mixing network to provide credit assignment across agents solving distinct subtasks, guided by a TD loss that fuses external rewards, intrinsic MI-based rewards, and predicted future rewards. Empirical results on StarCraft II SMAC II show SMAUG achieving superior performance and stability over baselines on hard and super-hard maps, with rapid early reward growth and robust behavior across seeds and maps. The work highlights the effectiveness of real-time subtask adaptation, multi-window trajectory encoding, and information-theoretic exploration in enabling flexible, scalable coordination in dynamic multi-agent environments.

Abstract

Instead of making behavioral decisions directly from the exponentially expanding joint observational-action space, subtask-based multi-agent reinforcement learning (MARL) methods enable agents to learn how to tackle different subtasks. Most existing subtask-based MARL methods are based on hierarchical reinforcement learning (HRL). However, these approaches often limit the number of subtasks, perform subtask recognition periodically, and can only identify and execute a specific subtask within the predefined fixed time period, which makes them inflexible and not suitable for diverse and dynamic scenarios with constantly changing subtasks. To break through above restrictions, a \textbf{S}liding \textbf{M}ultidimensional t\textbf{A}sk window based m\textbf{U}ti-agent reinforcement learnin\textbf{G} framework (SMAUG) is proposed for adaptive real-time subtask recognition. It leverages a sliding multidimensional task window to extract essential information of subtasks from trajectory segments concatenated based on observed and predicted trajectories in varying lengths. An inference network is designed to iteratively predict future trajectories with the subtask-oriented policy network. Furthermore, intrinsic motivation rewards are defined to promote subtask exploration and behavior diversity. SMAUG can be integrated with any Q-learning-based approach. Experiments on StarCraft II show that SMAUG not only demonstrates performance superiority in comparison with all baselines but also presents a more prominent and swift rise in rewards during the initial training stage.

Figures (7)

• Figure 1: The architecture of SMAUG. The green part denotes the inference network, the red part signifies the subtask-oriented policy network and the yellow part represents the mixing network. The concatenated trajectory set contains the trajectories of different window sizes concatenated with current and predicted trajectory segments. (1) The inference network iteratively infers the observations and rewards of future 1 to m time steps based on the subtask-oriented policy network, the current observation, and actions to predict trajectories. (2) The subtask-oriented policy network takes the concatenated trajectory set to produce local action values, $\{Q_i(\tau_i^t,a_i^t)\}$. (3) The mixing network utilizes $\{Q_i(\tau_i^t,a_i^t)\}$ to generate the overall action-value function $Q_{total}(\tau^t,\textbf{a}^t|\textbf{z}^t)$.The mixing network derives its hyperparameters from the current state and subtask set of agents $\{z_{i,k}^{t}\}$
• Figure 2: The process of subtask recognition utilizing the sliding multidimensional task window.
• Figure 3: The process of subtask prediction based on the inference network.
• Figure 4: Overall reward design diagram
• Figure 5: Performance comparison between SMAUG and other baselines in hard maps