SwarmRL: Building the Future of Smart Active Systems

TL;DR

SwarmRL addresses barriers to micro-scale robotic control by providing an open-source, high-performance Python package that unifies classical control and deep reinforcement learning within simulation and real-experiment workflows. It fuses active-matter modeling based on overdamped Langevin dynamics with actor-critic reinforcement learning, enabling policy training through episodes using returns $G_t=\sum_{t'=t}^T \gamma^{t'-t} r_{t'}$ and advantage $A_t^\pi=G_t-V_t^\pi$. The architecture is modular and GPU-enabled via Flax/JAX, supporting heterogeneous agents, force functions, tasks, observables, intrinsic rewards, exploration strategies, and visualization through ZnVis, with PPO as a ready-made optimization method. This work aims to accelerate cross-disciplinary micro-robotics research by lowering entry barriers and enabling scalable evaluation on HPC clusters while bridging simulations with experiments.

Abstract

This work introduces SwarmRL, a Python package designed to study intelligent active particles. SwarmRL provides an easy-to-use interface for developing models to control microscopic colloids using classical control and deep reinforcement learning approaches. These models may be deployed in simulations or real-world environments under a common framework. We explain the structure of the software and its key features and demonstrate how it can be used to accelerate research. With SwarmRL, we aim to streamline research into micro-robotic control while bridging the gap between experimental and simulation-driven sciences. SwarmRL is available open-source on GitHub at https://github.com/SwarmRL/SwarmRL.

Figures (3)

• Figure 1: Graphical overview of the RL workflow. The agent selects an action, $a_{t}$, based in its current state, $s_{t}$, and acts within the environment. This action yields a new state, $s_{t+1}$ and an associated reward, $r_{t+1}$.
• Figure 2: Overview of the SwarmRL software architecture. Light green boxes on the right-hand side of the figure represent modules with default settings that can be adjusted by users but do not need to be, dark green boxes indicate settings that need to be included in a system definition, blue boxes correspond to those directly handling colloid intelligence and properties, the engine is in orange, and grey represents classes that talk to the engine.
• Figure 3: Example renderings using SwarmRL and the ZnVis visualization engine. (left) A snapshot from a reservoir computing experiment using a reinforcement learning controlled swarm. (center) Colloids are trained to perform chemotaxis in the same fashion as bacteria. Here an open-source bacteria 3d model file has been used inside of SwarmRL to convey realism. The mesh was created by Sketchfab artist andrewfrueh, whose explicit permission was requested and granted before being used in this publication. The model is licensed under creative commons 4.0. (right) Demonstration of colloids rotating a rod, controlled using an RL algorithm.