Neural MMO: A Massively Multiagent Game Environment for Training and Evaluating Intelligent Agents

TL;DR

Neural MMO introduces a persistent, MMORPG-inspired environment to study training and evaluation of intelligent agents in large, open-ended multiagent settings. Agents operate on tile-based maps with foraging and combat mechanics, learning via policy gradients with a value baseline in a scalable architecture. Key findings show that larger populations amplify exploration while multiple unshared policies drive niche formation, and that agent dependencies emerge as policies adapt to others. The platform is designed for open-source release to enable broad exploration of niche formation, coevolution, and curriculum-like effects in multiagent learning at scale.

Abstract

The emergence of complex life on Earth is often attributed to the arms race that ensued from a huge number of organisms all competing for finite resources. We present an artificial intelligence research environment, inspired by the human game genre of MMORPGs (Massively Multiplayer Online Role-Playing Games, a.k.a. MMOs), that aims to simulate this setting in microcosm. As with MMORPGs and the real world alike, our environment is persistent and supports a large and variable number of agents. Our environment is well suited to the study of large-scale multiagent interaction: it requires that agents learn robust combat and navigation policies in the presence of large populations attempting to do the same. Baseline experiments reveal that population size magnifies and incentivizes the development of skillful behaviors and results in agents that outcompete agents trained in smaller populations. We further show that the policies of agents with unshared weights naturally diverge to fill different niches in order to avoid competition.

Figures (14)

• Figure 1: Our Neural MMO platform provides a procedural environment generator and visualization tools for value functions, map tile visitation distribution, and agent-agent dependencies of learned policies. Baselines are trained with policy gradients over 100 worlds.
• Figure 2: Our platform includes an animated 3D client and a toolbox used to produce the visuals in this work. Agents compete for food and water while engaging in strategic combat. See the Neural MMO section for a brief overview and the Supplement for full details.
• Figure 3: Maximum population size at train time varies in (16, 32, 64, 128). At test time, we merge the populations learned in pairs of experiments and evaluate lifetimes at a fixed population size. Agents trained in larger populations always perform better.
• Figure 4: Population size magnifies exploration: agents spread out to avoid competition.
• Figure 5: Populations count (number of species) magnifies niche formation. Visitation maps are overlaid over the game map; different colors correspond to different species. Training a single population tends to produce a single deep exploration path. Training eight populations results in many shallower paths: populations spread out to avoid competition among species.