TIGER-MARL: Enhancing Multi-Agent Reinforcement Learning with Temporal Information through Graph-based Embeddings and Representations

TL;DR

TIGER-MARL addresses the challenge of capturing evolving inter-agent coordination in multi-agent reinforcement learning by constructing dynamic temporal graphs and applying a temporal attention mechanism to produce time-aware agent embeddings. By explicitly linking current interactions with historical ones through static, self-history, and neighbor-history connections, TIGER enables adaptive coordination under partial observability. Empirical results on Gather and Tag show that TIGER improves task performance and sample efficiency over strong value-decomposition and graph-based baselines, with ablations clarifying the impact of temporal depth and structural connectivity. The work demonstrates the practicality of temporal graph reasoning for scalable, robust MARL and provides open-source code for reproducibility.

Abstract

In this paper, we propose capturing and utilizing \textit{Temporal Information through Graph-based Embeddings and Representations} or \textbf{TIGER} to enhance multi-agent reinforcement learning (MARL). We explicitly model how inter-agent coordination structures evolve over time. While most MARL approaches rely on static or per-step relational graphs, they overlook the temporal evolution of interactions that naturally arise as agents adapt, move, or reorganize cooperation strategies. Capturing such evolving dependencies is key to achieving robust and adaptive coordination. To this end, TIGER constructs dynamic temporal graphs of MARL agents, connecting their current and historical interactions. It then employs a temporal attention-based encoder to aggregate information across these structural and temporal neighborhoods, yielding time-aware agent embeddings that guide cooperative policy learning. Through extensive experiments on two coordination-intensive benchmarks, we show that TIGER consistently outperforms diverse value-decomposition and graph-based MARL baselines in task performance and sample efficiency. Furthermore, we conduct comprehensive ablation studies to isolate the impact of key design parameters in TIGER, revealing how structural and temporal factors can jointly shape effective policy learning in MARL. All codes can be found here: https://github.com/Nikunj-Gupta/tiger-marl.

Figures (8)

• Figure 1: Evolving coordination in MARL. Temporal interaction patterns can reveal useful relational cues to enhance learning.
• Figure 2: Dynamic temporal graph construction in tiger. At each timestep $t$, GAT-pruned structural edges (static neighbors) are augmented with self- and neighbor-history links from the past $K_{\mathrm{past\_self}}$ and $K_{\mathrm{past\_nbr}}$ steps, capturing both current and evolving coordination among agents.
• Figure 3: Overview of tiger. At each timestep, tiger constructs a dynamic temporal graph capturing both current and evolving inter-agent dependencies. Temporal attention then encodes these graphs into rich, weighted agent embeddings that integrate the most informative relational signals. These time-aware agent embeddings guide the learning of agent policies in MARL.
• Figure 4: Performance of tiger on Gather and Tag. Plots show task performance (win rate for Gather, return for Tag) over training steps, with standard deviation as shaded regions. (a) Mixer-based results showing faster convergence and stronger asymptotic performance with tiger-MIX. (b) Graph-based results showing more stable learning and higher final returns with tiger-DICG.
• Figure 5: Ablation studies on tiger. We vary (a) static connectivity ($K_{\mathrm{stat\_nbr}}$), (b) self-history depth ($K_{\mathrm{past\_self}}$), and (c) neighbor-history depth ($K_{\mathrm{past\_nbr}}$) to analyze how structural and temporal factors affect tiger’s performance.