Action-Graph Policies: Learning Action Co-dependencies in Multi-Agent Reinforcement Learning

TL;DR

This paper proposes Action Graph Policies (AGP), that model dependencies among agents' available action choices that enable agents to condition their decisions on global action dependencies and shows that AGP consistently outperforms these baselines in diverse multi-agent environments.

Abstract

Coordinating actions is the most fundamental form of cooperation in multi-agent reinforcement learning (MARL). Successful decentralized decision-making often depends not only on good individual actions, but on selecting compatible actions across agents to synchronize behavior, avoid conflicts, and satisfy global constraints. In this paper, we propose Action Graph Policies (AGP), that model dependencies among agents' available action choices. It constructs, what we call, \textit{coordination contexts}, that enable agents to condition their decisions on global action dependencies. Theoretically, we show that AGPs induce a strictly more expressive joint policy compared to fully independent policies and can realize coordinated joint actions that are provably more optimal than greedy execution even from centralized value-decomposition methods. Empirically, we show that AGP achieves 80-95\% success on canonical coordination tasks with partial observability and anti-coordination penalties, where other MARL methods reach only 10-25\%. We further demonstrate that AGP consistently outperforms these baselines in diverse multi-agent environments.

Figures (6)

• Figure 1: Illustration of Action-Graph Policies (AGP). The global action graph $\mathcal{G}_{\mathcal{A}}$ contains one node $u_a^i$ for each available action $a \in \mathcal{A}_i$ of every agent $i$, with edges representing learned coordination dependencies across action choices. Coordination contexts $\boldsymbol{\kappa}_i$ are constructed by aggregating information from this action graph prior to decentralized action selection.
• Figure 2: Coordination matrix games.(a) Top-$K$ Selection ($N{=}6$, $K{=}2$): baselines cluster near the independent-execution bound, while AGP achieves near-optimal coordination. (b) Top-$K$ with anti-coordination penalty: baselines suffer from relative overgeneralization, whereas AGP remains robust.
• Figure 3: Ablations. Removing cross-agent action dependencies or the action graph drastically collapses the performance, while a policy-gradient AGP variant recovers similar performance.
• Figure 4: Performance on multi-agent particle environments. AGP achieves the highest final test return across all six tasks, outperforming all MARL baselines. Shaded regions indicate $\pm$ one standard deviation over five seeds.
• Figure 5: Top-$K$ Selection: attention over the action graph. We visualize all attention heads and their mean at the final checkpoint. Attention exhibits structured cross-agent coupling (off-diagonal mass) rather than purely local/self-attention, supporting the interpretation that AGP constructs coordination contexts through action-level dependencies.

Theorems & Definitions (10)

• Theorem 3.1: Independent policies are not universally optimal
• Theorem 3.2: Value--policy mismatch
• Theorem 4.1: Strict Joint-Policy Expressivity
• Theorem 4.2: KL-Optimal Approximation of Centralized Joint Policies
• proof
• proof
• Theorem 2.1: Pairwise value factorization is not universally sufficient
• proof
• Theorem 3.1: KL Separation from Independent Execution
• proof