Aquarium: A Comprehensive Framework for Exploring Predator-Prey Dynamics through Multi-Agent Reinforcement Learning Algorithms

TL;DR

Aquarium presents a unified, open-source MARL environment for predator–prey research, integrating the PettingZoo framework to support reproducible, scalable experiments in a continuous 2D toroidal world. It delivers fully configurable dynamics, perception, and rewards, along with visualization and video recording to facilitate both quantitative benchmarking and qualitative analysis. Using PPO with GAE, the authors compare Individual Learning and Parameter Sharing policies, showing that shared-parameter training improves coordination and sample efficiency relative to independently trained agents, while strong baselines like TurnAway can outperform learned strategies in certain metrics. The work aims to standardize predator–prey MARL experimentation, enabling systematic investigations of swarm behaviors, social dilemmas, and emergent coordination with practical implications for multi-agent system design and ecological modelling.

Abstract

Recent advances in Multi-Agent Reinforcement Learning have prompted the modeling of intricate interactions between agents in simulated environments. In particular, the predator-prey dynamics have captured substantial interest and various simulations been tailored to unique requirements. To prevent further time-intensive developments, we introduce Aquarium, a comprehensive Multi-Agent Reinforcement Learning environment for predator-prey interaction, enabling the study of emergent behavior. Aquarium is open source and offers a seamless integration of the PettingZoo framework, allowing a quick start with proven algorithm implementations. It features physics-based agent movement on a two-dimensional, edge-wrapping plane. The agent-environment interaction (observations, actions, rewards) and the environment settings (agent speed, prey reproduction, predator starvation, and others) are fully customizable. Besides a resource-efficient visualization, Aquarium supports to record video files, providing a visual comprehension of agent behavior. To demonstrate the environment's capabilities, we conduct preliminary studies which use PPO to train multiple prey agents to evade a predator. In accordance to the literature, we find Individual Learning to result in worse performance than Parameter Sharing, which significantly improves coordination and sample-efficiency.

Figures (12)

• Figure 1: The Multi-Agent Reinforcement Learning Cycle (c.f. terry_multi-agent_2023). Within the aquarium environment of $n$ agents, each optimizes its policy to maximize individual rewards, while concurrently influencing the observations and rewards of others.
• Figure 2: Examples of the Aquarium With 16 Prey (Fishes) and One Predator (Shark) Agent.
• Figure 3: Calculation of the Steering Force. A limited fraction of the steering force is applied to smoothly transition from the current to the desired velocity, which is capped by the agent's maximum speed.
• Figure 4: Angle and Distance Criteria. An agent is considered within the field of view only if both conditions are met: staying within a defined angular range, $\alpha$ (left), and not exceeding a specified distance, $v$ (middle). Both conditions are fulfilled (right).
• Figure 5: Distance and Field of View Calculation in a Torus.