Dynamic Incentivized Cooperation under Changing Rewards

TL;DR

This work tackles cooperation in multi-agent systems when environmental rewards drift, a scenario where fixed peer-incentive schemes fail. It introduces DRIVE, a decentralized framework where agents reciprocally exchange reward differences to dynamically shape incentives, leveraging a TD-gated request/response protocol and a DRIVE-shaped reward to promote mutual cooperation. Theoretical results show that DRIVE aligns incentives in the Prisoner’s Dilemma, rendering cooperation a dominant strategy, and remains invariant to affine reward changes under per-epoch normalization. Empirically, DRIVE demonstrates robust cooperation across iterated and sequential social dilemmas under reward drift, outperforming state-of-the-art PI methods and maintaining performance where fixed-incentive methods falter. Overall, DRIVE offers a scalable, protocol-driven approach to sustaining cooperation in evolving environments, with graceful degradation under partial non-adherence and clear avenues for extending to more complex network topologies and adversarial settings.

Abstract

Peer incentivization (PI) is a popular multi-agent reinforcement learning approach where all agents can reward or penalize each other to achieve cooperation in social dilemmas. Despite their potential for scalable cooperation, current PI methods heavily depend on fixed incentive values that need to be appropriately chosen with respect to the environmental rewards and thus are highly sensitive to their changes. Therefore, they fail to maintain cooperation under changing rewards in the environment, e.g., caused by modified specifications, varying supply and demand, or sensory flaws - even when the conditions for mutual cooperation remain the same. In this paper, we propose Dynamic Reward Incentives for Variable Exchange (DRIVE), an adaptive PI approach to cooperation in social dilemmas with changing rewards. DRIVE agents reciprocally exchange reward differences to incentivize mutual cooperation in a completely decentralized way. We show how DRIVE achieves mutual cooperation in the general Prisoner's Dilemma and empirically evaluate DRIVE in more complex sequential social dilemmas with changing rewards, demonstrating its ability to achieve and maintain cooperation, in contrast to current state-of-the-art PI methods.

Figures (10)

• Figure 1: Motivating example on the reward sensitivity of PI approaches in a 2-player Prisoner’s Dilemma: IA requires careful tuning even under the original payoffs and fails to achieve cooperation in both scenarios. LIO achieves high cooperation under the original scale (left) but degenerates when the payoffs change, even though the inequalities for greed and fear (Eq. \ref{['eq:PD_inequalities']}) still hold. In contrast, DRIVE maintains robust cooperation across both settings without retuning.
• Figure 2: (a) Social dilemma payoff matrix with $R$, $P$, $T$, and $S$. In Prisoner's Dilemmas (PD), $T > R > P > S$ holds axelrod1984evolutionmacy2002learning. (b) A PD instance satisfying these inequalities.
• Figure 3: DRIVE exchange scheme. (a) If $\textit{TD}_1(\hat{u}_{t,1}) \geq 0$ (Eq. \ref{['eq:mi_td']}), agent 1 sends its reward $\hat{u}_{t,1}$ to neighbor agent 2 as a request. (b) Agent 2 calculates the difference $\Delta_{t,1,2}$ between its own average reward $\overline{u}_i$ in the current epoch $m$ and the received request $\hat{u}_{t,i}$, sent back as a response and used to shape the rewards of both agents (Eq. \ref{['eq:DRIVE_reward']}).
• Figure 4: Modified PD payoff matrices of different PI methods with payoff modifications highlighted in red hughes2018inequityyang2020learningphanJAAMAS2024.
• Figure 5: Average progress of DRIVE and other baselines in SSDs without reward change. Shaded areas show the 95% confidence interval. The results of LOLA-PG and POLA-DiCE are from foerster2018learningzhao2022proximal.

Theorems & Definitions (10)

• Theorem 1
• definition 1: Reward Change Function
• Theorem 2
• Theorem 3
• Theorem 4
• definition 2: Graphical PD
• proposition 1: Pairwise incentive alignment
• lemma 1: Affine transformations preserve PD structure
• lemma 2
• lemma 3: Scaling-invariance of normalized policy gradients