Evolutionary Cooperation with Game Transitions via Markov Decision Chain in Networked Population

TL;DR

This work develops a Markov decision chain framework that couples strategy evolution with environment-driven game transitions in networked populations. By allowing strategies to modulate transition rates and by enabling multi-step policy updates through simulated neighbor interactions, the model creates a bidirectional feedback loop between actions and environments. Simulations reveal that higher transition rates and larger environmental disparities promote cooperation even when the traditional benefit-to-cost condition is not satisfied, offering simulation-based guidance for coordinating multi-agent systems. The approach suggests practical implications for swarm intelligence and highlights avenues for extending to broader games and more sophisticated decision-making.

Abstract

Individual cooperative strategy influences the surrounding dynamic population, which in turn affects cooperative strategy. To better model this phenomenon, we develop a Markov decision chain based game transitions model and examine the dynamic transitions in game states of individuals within a network and their impact on the strategy's evolution. Additionally, we extend single-round strategy imitation to multiple rounds to better capture players' potential non-rational behavior. Using intensive simulations, we explore the effects of transition probabilities and game parameters on game transitions and cooperation. Our study finds that strategy-driven game transitions promote cooperation, and increasing the transition rates of Markov decision chains can significantly accelerate this process. By designing different Markov decision chains, these results provide simulation based guidance for practical applications in swarm intelligence, such as strategic collaboration.

Figures (6)

• Figure 1: Markov decision evolutionary model with simulated game. This figure illustrates a possible example of strategy and state evolution, involving a Markov process composed of three game states $G_1$, $G_2$, and $G_3$. The green and red arrows represent the transition probabilities under cooperation and defection strategies, respectively. Taking agent $i$ as an example, this agent first engages in a simulated game to reserve strategies until its policy is fully constructed. Subsequently, in the real game, the agent will apply strategies based on the policy.
• Figure 2: The effect of $\sigma$ and $\delta$ on the proportion of different game states. In (\ref{['fig2a']}), circles, squares, and triangles represent the density of three game states ${G_1, G_2, G_3}$ as $\sigma$ increases. The other parameters are $\delta = 0.1, \gamma=0 , b_1 = 3, \Delta = 1$ and $c = 1$. In (\ref{['fig2b']}), solid lines with circles, dashed lines with squares, and dotted lines with triangles correspond to $\sigma$ values of $0.4$, $0.5$, and $0.6$, respectively. According to the legend, different colors represent the densities for game states ${G_1, G_2, G_3}$ as $\delta$ varies. The other parameters are $b_1 = 3$, $\Delta = 1$, and $c = 1$. The simulation results are obtained after $10^3$ iterations on the WS network.
• Figure 3: The impact of $\sigma$ on the evolution of different game states. Panels show the time evolution of the density of game states ${G_1, G_2, G_3}$ at different $\sigma$ values, as indicated. Parameters are $\delta = 0.1$, $b_1 = 3$, $\Delta = 1$, and $c = 1$.
• Figure 4: The effect of game parameters on the proportion of game states. (\ref{['G1:A']}): we fix $\Delta=1$ to explore the effect of $c$ and $b_1$ on the density of $G_1$. (\ref{['G1:B']}): we fix $c=1$ to explore the effect of $b_1$ and $\Delta$ on the density of $G_1$. The other parameters are $\sigma = 0.5$, $\delta = 0.1$, and $\gamma= 0$. For the effect of game parameters on the proportion of $G_2$, we set $\sigma = 0.5$, $\delta = 0.1$, and $\gamma= 0$. Furthermore, $\Delta=1$ in panel (\ref{['G2:A']}), while $c=1$ in panel (\ref{['G2:B']}). Note that there is only a tiny difference between the minimal and maximal values in these panels. For the effect of game parameters on the proportion of $G_3$ we use $\sigma = 0.5$, $\delta = 0.1$, and $\gamma= 0$. In panel (\ref{['G3:A']}) we fix $\Delta=1$, while in panel (\ref{['G3:B']}) $c=1$ is used.
• Figure 5: The effect of transition parameters and policy size on the cooperation frequency. In (\ref{['fig.5a']}), we set the payoff parameters at $b_1 = 3$, $\Delta = 1$, and $\gamma = 0$. The curves with circular, square, triangular, rhombic, and pentagonal illustrate how the density of Cooperation in the population varies with $\sigma$ for $\delta$ values of $0.00$, $0.02$, $0.04$, $0.06$, and $0.08$. In (\ref{['fig.5b']}), we select three different values of $\sigma=\{0.3, 0.4, 0.5\}$, represented by curves with circles, squares, and triangles, respectively. The remaining parameters are $\delta=0.04$, $b_1=3$, $c=1$, and $\Delta=1$.

Theorems & Definitions (6)

• Definition 1
• Definition 2
• Proposition 1
• Proof
• Proposition 2
• Proof