How Exploration Breaks Cooperation in Shared-Policy Multi-Agent Reinforcement Learning

TL;DR

This study analyzes how exploration in shared-policy multi-agent reinforcement learning undermines cooperation in dynamic social dilemmas. Using a Prisoner's Dilemma on networks with a shared Deep Q-Network, the authors show that cooperation collapses as exploration increases, even when payoff-dominant equilibria exist. The collapse coincides with representational degradation and contraction of action-value gaps, indicating convergence to stable but low-cooperation attractors rather than learning instability. Mitigation via grouped policy learning and state augmentation delays collapse and offers guidance for designing robust, scalable MARL systems in social and economic settings.

Abstract

Multi-agent reinforcement learning in dynamic social dilemmas commonly relies on parameter sharing to enable scalability. We show that in shared-policy Deep Q-Network learning, standard exploration can induce a robust and systematic collapse of cooperation even in environments where fully cooperative equilibria are stable and payoff dominant. Through controlled experiments, we demonstrate that shared DQN converges to stable but persistently low-cooperation regimes. This collapse is not caused by reward misalignment, noise, or insufficient training, but by a representational failure arising from partial observability combined with parameter coupling across heterogeneous agent states. Exploration-driven updates bias the shared representation toward locally dominant defection responses, which then propagate across agents and suppress cooperative learning. We confirm that the failure persists across network sizes, exploration schedules, and payoff structures, and disappears when parameter sharing is removed or when agents maintain independent representations. These results identify a fundamental failure mode of shared-policy MARL and establish structural conditions under which scalable learning architectures can systematically undermine cooperation. Our findings provide concrete guidance for the design of multi-agent learning systems in social and economic environments where collective behavior is critical.

Figures (9)

• Figure 1: Cooperation levels in shared-policy DQN as a function of exploration strength $B$ and payoff harshness $D_r$. Each cell is the mean over 30 random seeds during evaluation. Cooperation degrades systematically as either parameter increases, forming an extended low-cooperation region in the upper-right. The smooth boundaries indicate structured collapse rather than sporadic failures.
• Figure 2: Collapse thresholds $D_r^*(B)$ as a function of exploration strength $B$ (mean over 30 seeds). (a) Shared DQN: all agents use one network. Threshold decreases monotonically with $B$ (identified at cooperation = 0.55). (b) Grouped DQN: 10 groups of 90 agents, each with independent network. Threshold shows non-monotonic dependence with partial recovery at high $B$ (identified at cooperation = 0.15). Different thresholds reflect different baseline cooperation levels. Dashed lines mark tested $D_r$ range $[0.10, 0.40]$. Hollow markers indicate threshold outside tested range.
• Figure 3: Effect of state augmentation on cooperation outcomes in shared-policy DQN under varying exploration strength $B$. All curves show mean cooperation levels averaged over 30 independent random seeds. The baseline curve corresponds to the original shared DQN, whose state encodes only local interaction information, consisting of each agent’s own recent action together with the recent actions of its neighbors. We compare this baseline against augmented variants that append additional scalar signals to the original state vector: a temporal exploration signal $\tau$, a coarse training-progress (annealing) indicator, and their joint inclusion. All configurations employ identical learning algorithms, network architectures, reward structures, and interaction topologies; only the information available to the shared value function differs. Joint augmentation yields the most robust behavior relative to the baseline, exhibiting a stable and weakly increasing cooperation trend as exploration strength $B$ increases, while substantially reducing sensitivity to exploration-induced degradation. In contrast, single-signal augmentation produces more limited and regime-dependent effects.
• Figure 4: Visualization of hidden-layer representations learned by the shared-policy DQN under increasing exploration strength $B$. Each panel shows a two-dimensional UMAP projection of hidden activations corresponding to agents’ states, with points colored by the selected action (cooperate or defect). At low exploration, cooperative and defective actions occupy partially distinct regions in the projected latent space. At intermediate $B$, the representations exhibit increased geometric clusterability, while under strong exploration the latent space becomes increasingly diffuse, with cooperative representations shrinking and losing stability. Silhouette scores, computed from unsupervised two-cluster partitioning of the hidden representations, quantify this non-monotonic evolution in cluster structure. Importantly, silhouette scores reflect geometric clusterability rather than action-discriminative separability, and their transient increase at intermediate $B$ does not correspond to improved support for cooperative decision-making.
• Figure 5: Average action-value statistics of the shared-policy DQN as functions of the exploration strength $B$, evaluated during a held-out evaluation phase with learning disabled, with the defection loss fixed at $D_r = 0.25$. All statistics are averaged over 30 independent random seeds. (a) Average action-value magnitude (Q-mean), computed as the mean absolute action value averaged over actions, agents, and encountered states. (b) Average action-value gap (Q-gap), defined as the mean absolute difference between the estimated Q-values of cooperative and defective actions. As exploration increases, the Q-gap decreases monotonically toward zero, while the Q-mean remains finite and bounded, indicating a contraction of relative value differences rather than unstable or divergent learning dynamics.