Emergence of Collective Open-Ended Exploration from Decentralized Meta-Reinforcement Learning

TL;DR

The paper addresses how collective open-ended exploration can emerge in decentralized multi-agent meta-reinforcement learning under an open-ended task distribution. It introduces a procedurally generated environment that builds task trees from five subtasks and trains two independent recurrent-policy agents with PPO, employing a mix of single- and multi-agent episodes to alleviate credit assignment issues. The results show strong generalization to novel objects and coordination-heavy tasks, and extend to open-ended task trees up to depth $d=6$, indicating robust collective exploration capabilities. The work provides open-source code and videos, highlighting the potential of decentralized coordination in complex, open-ended domains.

Abstract

Recent works have proven that intricate cooperative behaviors can emerge in agents trained using meta reinforcement learning on open ended task distributions using self-play. While the results are impressive, we argue that self-play and other centralized training techniques do not accurately reflect how general collective exploration strategies emerge in the natural world: through decentralized training and over an open-ended distribution of tasks. In this work we therefore investigate the emergence of collective exploration strategies, where several agents meta-learn independent recurrent policies on an open ended distribution of tasks. To this end we introduce a novel environment with an open ended procedurally generated task space which dynamically combines multiple subtasks sampled from five diverse task types to form a vast distribution of task trees. We show that decentralized agents trained in our environment exhibit strong generalization abilities when confronted with novel objects at test time. Additionally, despite never being forced to cooperate during training the agents learn collective exploration strategies which allow them to solve novel tasks never encountered during training. We further find that the agents learned collective exploration strategies extend to an open ended task setting, allowing them to solve task trees of twice the depth compared to the ones seen during training. Our open source code as well as videos of the agents can be found on our companion website.

Figures (7)

• Figure 1: Task Tree Sampling and Episode Rollout.A) shows the task tree sampling process. First three subtasks are sampled from the distribution of subtasks (Section \ref{['task_types']}), one for each stage of the task tree. All of the objects required to solve the subtask for stage one and some of the objects required by subtasks in later stages are then placed in the environment. The remaining objects required to solve the later subtasks can be created through solving preceding subtasks (Section \ref{['task_sampling']}). B) shows an example of a single episode rollout. The agents have to complete the subtasks sequentially in order to create objects which are needed by the subtasks in later stages. Since a new task tree with different subtasks is sampled at the beginning of each episode and no information about the subtasks is given to the agents, the agents have to explore the environment and interact with all present objects so solve the subtask at each stage. Videos of the agents behaviors can be found on https://sites.google.com/view/collective-open-ended-explore.
• Figure 2: Success rates for $100$% vs $50$% multi agent episodes during training
• Figure 3: Mean stage $3$ success rate difference between two agents trained on $100$% vs $50$% multi agent episodes
• Figure 4: Mean stage $3$ success rate on single agent episodes for agents trained on $100$% vs $50$% multi agent episodes
• Figure 5: Success rates with novel objects