Ant-inspired Walling Strategies for Scalable Swarm Separation: Reinforcement Learning Approaches Based on Finite State Machines

TL;DR

The paper tackles the challenge of maintaining spatial separation in heterogeneous robotic swarms performing concurrent tasks by introducing dissipative infrastructures inspired by army ant walling. It develops two decentralized controllers: a hand-crafted FSM-based walling mechanism and an FSM–DQN hybrid that learns when to switch between walling, avoidance, and exploration using UWB/AoA sensing with an attention-based multi-head architecture. In simulation, both approaches reduce inter-swarm mixing, with the RL-enhanced controller achieving 40–50% reductions and faster convergence, demonstrating scalable, sensor-robust decentralized coordination. The work argues that RL-based state switching within a structured FSM framework provides an explainable, practical alternative to fully end-to-end deep RL for real-world swarm deployments.

Abstract

In natural systems, emergent structures often arise to balance competing demands. Army ants, for example, form temporary "walls" that prevent interference between foraging trails. Inspired by this behavior, we developed two decentralized controllers for heterogeneous robotic swarms to maintain spatial separation while executing concurrent tasks. The first is a finite-state machine (FSM)-based controller that uses encounter-triggered transitions to create rigid, stable walls. The second integrates FSM states with a Deep Q-Network (DQN), dynamically optimizing separation through emergent "demilitarized zones." In simulation, both controllers reduce mixing between subgroups, with the DQN-enhanced controller improving adaptability and reducing mixing by 40-50% while achieving faster convergence.

Figures (11)

• Figure 1: Illustration of example "dissipative infrastructures" that enforce self-organized separation in collectives. (a) Simulation snapshot showing populations of two types of agents (orange, black) that maintain separation by following one of the controllers presented in this paper, resulting in the emergence of a dynamic "virtual wall" (dotted line) between the populations. (b) Inspiration from observations in Baudier and Pavlic baudier2020incidental of self-organized walls of army ants that act to prevent two large trails from intersecting.
• Figure 2: Finite state machine (FSM)-based robot controller.
• Figure 3: Reinforcement learning-based hybrid controller architecture: The diagram illustrates the RL-based navigation system, which processes multi-modal sensor inputs from neighboring robots, including UWB distance measurements, Angle of Arrival (AoA) data, and binary swarm-type indicators. The architecture employs a feature extraction module with a linear transformation and ReLU activation, followed by a multi-head attention mechanism for efficient processing of variable-sized input data. The final Q-value output layer generates optimal navigation actions, enabling adaptive state switching between walling, avoidance, and exploration behaviors to maintain swarm coordination and spatial separation.
• Figure 4: Top row: Initial robot positions (step 1) in five simulated cases: (Case 1) Robots initialized at the right and left edges of the environment; (Case 2) Orange robots initialized at 1/8 the environment width from the left edge, black robots at the right edge; (Case 3) Robots randomly initialized throughout the environment; (Case 4) Robots initialized in concentric circles (black: inner, orange: outer); (Case 5) Robots randomly initialized near the center of the environment. Middle row: Robot positions at step 1000 under the SM controller. Bottom row: Robot positions at step 1000 under the RL controller.
• Figure 5: Case 1: Coverage and mixing ratio percentages over time (simulation steps) for (top left) SM controller with walling $\text{timer} = 0\,\text{s}$; (top right) SM controller with $\text{timer} = 3\,\text{s}$; (bottom left) RL controller with $\text{timer} = 0\,\text{s}$; (bottom right) RL controller with $\text{timer} = 3\,\text{s}$. Solid lines (shaded regions) are stepwise averages (ranges) over 100 runs.