Steady-State Strategy Synthesis for Swarms of Autonomous Agents

TL;DR

This work extends steady-state policy synthesis from single to multi-agent MDPs, formalizing long-run color-based frequency objectives and analyzing the relative power of memoryless, full memoryless, finite-memory, and history-dependent strategies. It establishes strong complexity barriers: existence of feasible frequency vectors is NP-hard (even for simple profiles) and PSPACE-hard under colored variants, while evaluating some profile types can be intractable; nonetheless, the authors derive a polynomial-space decidable approach for fixed numbers of agents in the FR$_m$ setting. To address scalability, they propose an incremental LP-based synthesis algorithm that builds full memoryless profiles by adding agents one at a time, keeping LP sizes independent of the agent count and enabling practical synthesis on large instance classes. Empirical evaluation on randomly generated aperiodic and periodic graphs demonstrates that the incremental method often reduces the required number of agents and outperforms naive strategy-sharing baselines, albeit with higher per-benchmark computation. Overall, the paper balances deep theoretical hardness results with a scalable, experimentally validated synthesis technique applicable to multi-agent steady-state constraints in MDPs.

Abstract

Steady-state synthesis aims to construct a policy for a given MDP $D$ such that the long-run average frequencies of visits to the vertices of $D$ satisfy given numerical constraints. This problem is solvable in polynomial time, and memoryless policies are sufficient for approximating an arbitrary frequency vector achievable by a general (infinite-memory) policy. We study the steady-state synthesis problem for multiagent systems, where multiple autonomous agents jointly strive to achieve a suitable frequency vector. We show that the problem for multiple agents is computationally hard (PSPACE or NP hard, depending on the variant), and memoryless strategy profiles are insufficient for approximating achievable frequency vectors. Furthermore, we prove that even evaluating the frequency vector achieved by a given memoryless profile is computationally hard. This reveals a severe barrier to constructing an efficient synthesis algorithm, even for memoryless profiles. Nevertheless, we design an efficient and scalable synthesis algorithm for a subclass of full memoryless profiles, and we evaluate this algorithm on a large class of randomly generated instances. The experimental results demonstrate a significant improvement against a naive algorithm based on strategy sharing.

Figures (14)

• Figure 1: Left: A simple MDP with three non-deterministic vertices $v_1$, $v_2$, and $v_3$. Right: A memoryless strategy for a single agent can achieve the frequency vector $(0.5{-}\delta, 2\delta, 0.5{-}\delta)$ for an arbitrarily small $\delta{>}0$ by choosing a sufficiently small $\varepsilon {>} 0$. However, the frequency vector $(0.5,0,0.5)$ is achievable only by an infinite-memory strategy where the $\varepsilon$ is "progressively smaller" and approaches $0$ as the vertices $v_1$ and $v_3$ are revisited. Middle: A full memoryless strategy can achieve the frequency vector $(1{-}\delta_1{-}\delta_2,\delta_1,\delta_2)$ where $\delta_1{+}\delta_2>0$ is arbitrarily small by choosing a sufficiently small $\varepsilon{>}0$. However, the vector $(1,0,0)$ is achievable only by a (non-full) strategy assigning $1$ to the self-loop $v_1 {\to} v_1$.
• Figure 2: The structure of cyclic classes. For all states $s,t$ we have that $P(s,t) > 0$ only if $s \in S_i$ and $t \in S_{i{+}1 ~\mathrm{mod}~ d}$ for some $i<d$.
• Figure 3: The graphs $D_1, D_2$, and $D_3$.
• Figure 4: The linear program for $\textit{Obj},\pi,C,D_q = (V^q,E^q,p^q)$.
• Figure 5: Numbers of agents sufficient to satisfy the objective using each of the algorithms (lower is better). Each point $(x,y)$ is a benchmark for which the objective is satisfied by $x$ agents by the baseline algorithm, and $y$ agents by Algorithm \ref{['alg:inc-synthesis']}. Divided by the type of the graph (aperiodic/periodic).

Theorems & Definitions (7)

• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• Definition 1
• Proposition 1
• proof