Unreal-MAP: Unreal-Engine-Based General Platform for Multi-Agent Reinforcement Learning

TL;DR

Unreal-MAP provides an open-source Unreal Engine–based platform for general MARL, featuring a five-layer architecture that separates engine, task, and algorithm components and enables controllable time flow. It formalizes MARL as a Partially Observable Markov Game (POMG) described by the 8-tuple ⟨N, {S^i}, {O^i}, {Ω^i}, {A^i}, {T^i}, r, γ⟩, with team policies optimized via $\bar{π}^{*}_A = \arg\max_{\bar{π}_A} \mathbb{E}_{\bar{π}_A}[ \sum_{k=0}^{∞} γ^k \sum_{i in A} r^i_{t+k} | \bar{s}_t = \bar{s} ]$. The accompanying HMAP framework decouples Task-Core-Algorithm components to support multi-team training across Unreal-MAP and other MARL environments, enabling rapid deployment of built-in and third-party policies. Empirical results across 15 tasks and 7 algorithms reveal complementary strengths of value-based and actor-critic methods depending on reward structure and scale, while demonstrating high simulation throughput and a plausible path toward sim-to-real deployment via a hardware-in-the-loop setup. Overall, Unreal-MAP aims to accelerate MARL research by delivering scalable realism, flexible task customization, cross-framework compatibility, and practical sim-to-real capabilities.

Abstract

In this paper, we propose Unreal Multi-Agent Playground (Unreal-MAP), an MARL general platform based on the Unreal-Engine (UE). Unreal-MAP allows users to freely create multi-agent tasks using the vast visual and physical resources available in the UE community, and deploy state-of-the-art (SOTA) MARL algorithms within them. Unreal-MAP is user-friendly in terms of deployment, modification, and visualization, and all its components are open-source. We also develop an experimental framework compatible with algorithms ranging from rule-based to learning-based provided by third-party frameworks. Lastly, we deploy several SOTA algorithms in example tasks developed via Unreal-MAP, and conduct corresponding experimental analyses. We believe Unreal-MAP can play an important role in the MARL field by closely integrating existing algorithms with user-customized tasks, thus advancing the field of MARL.

Figures (9)

• Figure 1: The research workflow for using Unreal-MAP. For novice users, Unreal-MAP provides direct access to built-in tasks, and offers comprehensive algorithm deployment functions and result visualization capabilities. For advanced users, Unreal-MAP enables the modification of built-in tasks or the creation of new tasks to test research ideas.
• Figure 2: Architecture of Unreal-MAP. Unreal-MAP employs a hierarchical, five-layered architecture, all of which are open source. Users can modify all elements within POMG by configuring parameters through the Python-based interface layer. For more advanced development requirements, users can conveniently adjust scenario elements using Blueprint through the advanced module layer.
• Figure 3: Example scenarios and tasks of Unreal-MAP. Users can develop new scenarios using Unreal-MAP, and create a variety of MARL tasks by adjusting properties such as the number, types, and teams of agents.
• Figure 4: The comparison of test win rate for all tested algorithms across 15 tasks. The shadowed area depicts the 95% confidence interval.
• Figure 5: The impact of the number of parallel processes and the time dilation factor in Unreal-MAP on simulation efficiency and computational resource consumption.