Discovering Coordinated Joint Options via Inter-Agent Relative Dynamics

TL;DR

This paper tackles the challenge of discovering strongly coordinated multi-agent options under partial observability by introducing a Fermat-state–based inter-agent relative representation. It develops a multi-dimensional n-distance framework with a Fermat encoder and temporal distance metrics, coupled with a mutual-information–based disentanglement objective, to compress joint state space while preserving coordination-relevant structure. By building graph Laplacian eigenvectors on the relative state representation (ALLO) and using them as intrinsic rewards, the authors generate joint options that exhibit coordinated inter-agent alignment and timing. The approach is integrated into a MacDec-POMDP-like setting to execute joint options, and empirical results in Level-Based Foraging and Overcooked show improved downstream coordination and robustness over baselines and scalar representations, with insights into option counts and domain complexity. This work advances scalable coordination learning by emphasizing inter-agent relational dynamics and disentangled, multi-feature representations for multi-agent option discovery.

Abstract

Temporally extended actions improve the ability to explore and plan in single-agent settings. In multi-agent settings, the exponential growth of the joint state space with the number of agents makes coordinated behaviours even more valuable. Yet, this same exponential growth renders the design of multi-agent options particularly challenging. Existing multi-agent option discovery methods often sacrifice coordination by producing loosely coupled or fully independent behaviours. Toward addressing these limitations, we describe a novel approach for multi-agent option discovery. Specifically, we propose a joint-state abstraction that compresses the state space while preserving the information necessary to discover strongly coordinated behaviours. Our approach builds on the inductive bias that synchronisation over agent states provides a natural foundation for coordination in the absence of explicit objectives. We first approximate a fictitious state of maximal alignment with the team, the \textit{Fermat} state, and use it to define a measure of \textit{spreadness}, capturing team-level misalignment on each individual state dimension. Building on this representation, we then employ a neural graph Laplacian estimator to derive options that capture state synchronisation patterns between agents. We evaluate the resulting options across multiple scenarios in two multi-agent domains, showing that they yield stronger downstream coordination capabilities compared to alternative option discovery methods.

Figures (14)

• Figure 1: Option discovery on inter-agent relative representations for a state factorisation function $g$, Fermat encoder $\phi$, state distance encoder $d_\theta$ and a graph Laplacian eigenvector approximator $\mu$.
• Figure 2: The first three non-trivial eigenvectors of the graph Laplacian for an 15x15 grid environment with three agents, as the only entities in the grid, under varying state representations: single agent state spaces (left), raw joint state spaces (center) and inter-agent relative state representations (right). For visibility, we fix the position of two agents for the multi-agent scenarios (at [(1,4), (1,7)] and [(7,7), (7,8)]), and display the values when varying the position of the remaining agent.
• Figure 3: Policy roll-outs visualisation of the first four relative options in the 15×15 grid environment with four agents. Arrows indicate the actions taken by each agent’s policy, coloured circles mark the final states (before the termination action is triggered), and the white circle denotes the estimated Fermat state corresponding to these final states. The bars on the right of each figure show the Fermat $n$-distance estimates for each feature. Please see Appendix \ref{['appendix:option_policy_plots']} for other state initializations.
• Figure 4: Downstream task performance analysis for both environments (LBF on the left, Overcooked on the right). The top row compares IQL+IARO against option-free baseline algorithms, while the bottom row compares it against IQL augmented with other option discovery methods.
• Figure 5: Downstream task performance for the most complex scenario in LBF and Overcooked, evaluated using different numbers of options. We report IQM scores over 15 seeds and 64 evaluation episodes at the end of training for each configuration, with standard deviations shown as error bars.

Theorems & Definitions (3)

• Definition 1
• Proposition 1
• Proposition 2